anoncvs still slow

Michael Fuhr <mike@fuhr.org> writes:

anoncvs (svr4, 66.98.251.159) is still slow responding to "cvs update";
it's been spotty for about a week now. Tcpdump shows connections being
established but then long delays for ACKs, sometimes long enough for cvs
to time out. Any updates on what's going on?

Magnus apparently knows what the problem is:
http://archives.postgresql.org/pgsql-hackers/2006-05/msg01002.php
but I haven't seen any of the "other mails" he mentioned.

regards, tom lane

On Sat, 27 May 2006, Tom Lane wrote:

Michael Fuhr <mike@fuhr.org> writes:

anoncvs (svr4, 66.98.251.159) is still slow responding to "cvs update";
it's been spotty for about a week now. Tcpdump shows connections being
established but then long delays for ACKs, sometimes long enough for cvs
to time out. Any updates on what's going on?

Magnus apparently knows what the problem is:
http://archives.postgresql.org/pgsql-hackers/2006-05/msg01002.php
but I haven't seen any of the "other mails" he mentioned.

svr4 / anoncvs needs a major upgrade ... the problem is that the only part
of that vServer that I know nothing about is the bittorrent stuff, which,
in itself, needs an upgrade ... I sent a note to Magnus that, whenever
he's ready with the bittorrent stuff, I can do the rest of the upgrade, so
its in his court right now :)

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

anoncvs (svr4, 66.98.251.159) is still slow responding to "cvs
update"; it's been spotty for about a week now. Tcpdump shows
connections being established but then long delays for ACKs,
sometimes long enough for cvs to time out. Any updates on

what's going on?

Magnus apparently knows what the problem is:
http://archives.postgresql.org/pgsql-hackers/2006-05/msg01002.php
but I haven't seen any of the "other mails" he mentioned.

Right, those mails were sent in private to Marc, because they outline
some fairly severe (IMHO) configuration errors on svr4 and at least one
other postgresql.org mailserver, that is the main reason behind the
problems. I didn't want those details to go out public and be archived.

svr4 / anoncvs needs a major upgrade ... the problem is that
the only part of that vServer that I know nothing about is
the bittorrent stuff, which, in itself, needs an upgrade ...
I sent a note to Magnus that, whenever he's ready with the
bittorrent stuff, I can do the rest of the upgrade, so its in
his court right now :)

Um, *what*?

AFAICS, this is caused by the machine attempting to relay thousands and
thousands of spam emails (some quick checked showed a rate of about 1
spam / 5 seconds enytering the queue - and I know I deleted almost
20,000 from the queue)

This is a *configuration error*. if we *wanted* all this spam to be
relayed, it would be a performance problem. But to be efficient, the
spam has to be rejected *before* it enters the system. I've suggested a
couple of different things to be done to fix or at least decrease this
problem. From what I can tell, none have been implemented.

Now for the other problems, I propose the following:

For bittorrent, I propose we take it out. We've suggested it before, I
don't recall receiving any real requests to keep it, and IMHO it's way
much more pain than it's worth. Therefor, unless someone objects, I'll
pull the bittorrent links from the website in a couple of days, and then
we can just remove it from the server.

Also, if anoncvs is a problem that we need quickly fixed, can we mov eit
quickly to a different server. Say Ferengi, which has both bandwidth and
horsepower to spare in loads. Do we require some special-special version
of cvs, or just plain cvs?

//Magnus

Hi,

On Sun, 2006-05-28 at 21:25 +0200, Magnus Hagander wrote:

For bittorrent, I propose we take it out. We've suggested it before, I
don't recall receiving any real requests to keep it, and IMHO it's way
much more pain than it's worth. Therefor, unless someone objects, I'll
pull the bittorrent links from the website in a couple of days, and
then we can just remove it from the server.

Please go for it.
--
The PostgreSQL Company - Command Prompt, Inc. 1.503.667.4564
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: plPHP, plPerlNG - http://www.commandprompt.com/

For bittorrent, I propose we take it out. We've suggested it before, I
don't recall receiving any real requests to keep it, and IMHO it's way
much more pain than it's worth.

We received a couple of requests for the torrent on the IRC channel when
the update was released. Just FYI.

Therefor, unless someone objects, I'll

pull the bittorrent links from the website in a couple of days, and then
we can just remove it from the server.

Also, if anoncvs is a problem that we need quickly fixed, can we mov eit
quickly to a different server. Say Ferengi, which has both bandwidth and
horsepower to spare in loads. Do we require some special-special version
of cvs, or just plain cvs?

CMD has a spare machine we can host it on as well if you like.

Sincerely,

Joshua D. Drake

//Magnus

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Sun, 28 May 2006, Magnus Hagander wrote:

AFAICS, this is caused by the machine attempting to relay thousands and
thousands of spam emails (some quick checked showed a rate of about 1
spam / 5 seconds enytering the queue - and I know I deleted almost
20,000 from the queue)

And how exactly would you like me to fix *that*? The reason those were in
the queue is because svr4 is a legit MX record for the mailing lists ...
the messages are being delivered into svr4's mail queue, and
mail.postgresql.org subsequently refusing htem because they are for
invalid addresses ...

If I remove svr4 as an MX record, its just going to move to a different
machine ...

So, how exactly would you like me to "fix" that problem?

For bittorrent, I propose we take it out. We've suggested it before, I
don't recall receiving any real requests to keep it, and IMHO it's way
much more pain than it's worth. Therefor, unless someone objects, I'll
pull the bittorrent links from the website in a couple of days, and then
we can just remove it from the server.

That works for me ... let me know once its is down, and then I can easily
do the upgrade ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

AFAICS, this is caused by the machine attempting to relay thousands
and thousands of spam emails (some quick checked showed a rate of
about 1 spam / 5 seconds enytering the queue - and I know I deleted
almost 20,000 from the queue)

And how exactly would you like me to fix *that*? The reason
those were in the queue is because svr4 is a legit MX record
for the mailing lists ...
the messages are being delivered into svr4's mail queue, and
mail.postgresql.org subsequently refusing htem because they
are for invalid addresses ...

If I remove svr4 as an MX record, its just going to move to a
different machine ...

So, how exactly would you like me to "fix" that problem?

The complete fix is of course to apply the same ingress filtering on all
machines.

If that's not possible, do it as much as possible. As the email
addresses existing on svr1 is fairly static, it shouldn't be too hard to
teach svr4 (and other MXen if there are any) about them.

To make graylisting properly effective that also needs to be applied on
all entrypoints, otherwise svr4 will just solve the problems for the
spammers who have software that won't retry.

The quick fix is, as I wrote in one of my earlier mails, to configure
svr1 not to tell svr4 to *retry delivery*, but to just junk the mail
right away. It'll still cause joe-job style problems, but it won't load
up the queue for days.

For bittorrent, I propose we take it out. We've suggested

it before, I

don't recall receiving any real requests to keep it, and

IMHO it's way

much more pain than it's worth. Therefor, unless someone

objects, I'll

pull the bittorrent links from the website in a couple of days, and
then we can just remove it from the server.

That works for me ... let me know once its is down, and then
I can easily do the upgrade ...

I'll give it a day or two more for people to complain, and then junk it.

//Magnus

On Mon, 29 May 2006, Magnus Hagander wrote:

The quick fix is, as I wrote in one of my earlier mails, to configure
svr1 not to tell svr4 to *retry delivery*, but to just junk the mail
right away. It'll still cause joe-job style problems, but it won't load
up the queue for days.

But, from my look at the queue on svr4, this is already being done ... the
queue contains a bunch of MAILER-DAEMON bounces back for 'recipient
unknown', which is what is supposed to happen ...

but, your point about the greylisting makes sense ... will work on
implementing that one tonight ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

The quick fix is, as I wrote in one of my earlier mails, to

configure

svr1 not to tell svr4 to *retry delivery*, but to just junk

the mail

right away. It'll still cause joe-job style problems, but it won't
load up the queue for days.

But, from my look at the queue on svr4, this is already being
done ... the queue contains a bunch of MAILER-DAEMON bounces
back for 'recipient unknown', which is what is supposed to happen ...

That's because I've deleted thousands of emails already, and run the
delete script once every hour or so in order to keep it living.
(I bet your "mailq" command didn't take almost an hour - that's what it
did when I ran it this morning)

Run something like:
mailq | grep "Recipient address rejected"

This will currently show 283 emails, all backed to svr1.

To clean up the queue (of this type of emails only), run
mailq |./t.pl |postsuper -d -

from roots homedir.

The mails you are seeing are the ones generated after the other ones
have been sitting in the queue for a couple of days. They were also in
the thousands before, but since I try to cut down the queue at every
chance I get now, it usually doesn't get that far, so they don't
increase that much.

but, your point about the greylisting makes sense ... will
work on implementing that one tonight ...

Great.

//Magnus

On Mon, 29 May 2006, Magnus Hagander wrote:

The quick fix is, as I wrote in one of my earlier mails, to

configure

svr1 not to tell svr4 to *retry delivery*, but to just junk

the mail

right away. It'll still cause joe-job style problems, but it won't
load up the queue for days.

But, from my look at the queue on svr4, this is already being
done ... the queue contains a bunch of MAILER-DAEMON bounces
back for 'recipient unknown', which is what is supposed to happen ...

That's because I've deleted thousands of emails already, and run the
delete script once every hour or so in order to keep it living.
(I bet your "mailq" command didn't take almost an hour - that's what it
did when I ran it this morning)

Run something like:
mailq | grep "Recipient address rejected"

I thought that the above was supposed to be a perm error, not temp? Does
anyone know what I need to set in postfix on svr1 to change it to a perm?

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

The quick fix is, as I wrote in one of my earlier mails, to

configure

svr1 not to tell svr4 to *retry delivery*, but to just junk

the mail

right away. It'll still cause joe-job style problems, but

it won't

load up the queue for days.

But, from my look at the queue on svr4, this is already being done
... the queue contains a bunch of MAILER-DAEMON bounces back for
'recipient unknown', which is what is supposed to happen ...

That's because I've deleted thousands of emails already,

and run the

delete script once every hour or so in order to keep it living.
(I bet your "mailq" command didn't take almost an hour -

that's what

it did when I ran it this morning)

Run something like:
mailq | grep "Recipient address rejected"

I thought that the above was supposed to be a perm error, not
temp? Does anyone know what I need to set in postfix on svr1
to change it to a perm?

Yes, htat's what I sent before :-)

c) Change svr1 parameters to:
unknown_relay_recipient_reject_code = 550
and
unknown_local_recipient_reject_code = 550

//Magnus

On Mon, May 29, 2006 at 03:00:44PM -0300, Marc G. Fournier wrote:

Run something like:
mailq | grep "Recipient address rejected"

I thought that the above was supposed to be a perm error, not temp? Does
anyone know what I need to set in postfix on svr1 to change it to a perm?

Do you have soft bounce turned on? A mailbox unavailable message
should be a 550 error. Or is the problem that the relay gets back
the rejection, and then queues a message to the originating mailer
(which was, of course, a drone, and therefore won't be available)?
The latter case will _also_ eat up a ton of resources.
Unfortunately, there's no standards-compliant way to drop such
connections on the floor instead of erroring, once you've accepted
the mail.

Better to reject at the time of connection, which is what the
local_recipient_maps setting is for.

A

--
Andrew Sullivan | ajs@crankycanuck.ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland

Run something like:
mailq | grep "Recipient address rejected"

I thought that the above was supposed to be a perm error,

not temp?

Does anyone know what I need to set in postfix on svr1 to

change it to

a perm?

Yes, htat's what I sent before :-)

c) Change svr1 parameters to:
unknown_relay_recipient_reject_code = 550 and
unknown_local_recipient_reject_code = 550

By the way, the proper way to fix it it o use a relay_recipient_map. To
this map, add all the users that are valid in postgresql.org, and
install it on svr4. Then svr4 will reject the connections *before* it
queues up the mail, and it'll also get rid of the incorrect bounces.

//Magnus

On Mon, May 29, 2006 at 02:14:42PM -0300, Marc G. Fournier wrote:

On Sun, 28 May 2006, Magnus Hagander wrote:

AFAICS, this is caused by the machine attempting to relay thousands and
thousands of spam emails (some quick checked showed a rate of about 1
spam / 5 seconds enytering the queue - and I know I deleted almost
20,000 from the queue)

And how exactly would you like me to fix *that*? The reason those were in
the queue is because svr4 is a legit MX record for the mailing lists ...
the messages are being delivered into svr4's mail queue, and
mail.postgresql.org subsequently refusing htem because they are for
invalid addresses ...

If I remove svr4 as an MX record, its just going to move to a different
machine ...

So, how exactly would you like me to "fix" that problem?

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim C. Nasby wrote:

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

umm ... we allow non-subscribers to post, the posts just have to be
approved. How would we still do that?

cheers

andrew

Andrew Dunstan wrote:

Jim C. Nasby wrote:

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

umm ... we allow non-subscribers to post, the posts just have to be
approved. How would we still do that?

What's checked is the recipient, not the sender.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:

Andrew Dunstan wrote:

Jim C. Nasby wrote:

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

umm ... we allow non-subscribers to post, the posts just have to be
approved. How would we still do that?

What's checked is the recipient, not the sender.

ah, ok. sorry for the noise.

cheers

andrew

On Tue, 30 May 2006, Jim C. Nasby wrote:

On Mon, May 29, 2006 at 02:14:42PM -0300, Marc G. Fournier wrote:

On Sun, 28 May 2006, Magnus Hagander wrote:

AFAICS, this is caused by the machine attempting to relay thousands and
thousands of spam emails (some quick checked showed a rate of about 1
spam / 5 seconds enytering the queue - and I know I deleted almost
20,000 from the queue)

And how exactly would you like me to fix *that*? The reason those were in
the queue is because svr4 is a legit MX record for the mailing lists ...
the messages are being delivered into svr4's mail queue, and
mail.postgresql.org subsequently refusing htem because they are for
invalid addresses ...

If I remove svr4 as an MX record, its just going to move to a different
machine ...

So, how exactly would you like me to "fix" that problem?

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

The list of email addresses changes over time ... so whomever creates a
new mailbox would need to remember to add it on the MX servers as well ...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664

On Tue, May 30, 2006 at 02:01:07PM -0400, Andrew Dunstan wrote:

Jim C. Nasby wrote:

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

umm ... we allow non-subscribers to post, the posts just have to be
approved. How would we still do that?

I'm assuming we're talking about a list of valid To: addresses, not From:
addresses. That list should be fairly short...

Have a nice day,
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/

On Tue, May 30, 2006 at 04:21:46PM -0300, Marc G. Fournier wrote:

On Tue, 30 May 2006, Jim C. Nasby wrote:

On Mon, May 29, 2006 at 02:14:42PM -0300, Marc G. Fournier wrote:

On Sun, 28 May 2006, Magnus Hagander wrote:

AFAICS, this is caused by the machine attempting to relay thousands and
thousands of spam emails (some quick checked showed a rate of about 1
spam / 5 seconds enytering the queue - and I know I deleted almost
20,000 from the queue)

And how exactly would you like me to fix *that*? The reason those were in
the queue is because svr4 is a legit MX record for the mailing lists ...
the messages are being delivered into svr4's mail queue, and
mail.postgresql.org subsequently refusing htem because they are for
invalid addresses ...

If I remove svr4 as an MX record, its just going to move to a different
machine ...

So, how exactly would you like me to "fix" that problem?

Postfix allows you to specify a list of valid email addresses. It should
be a simple matter of specifying what all the valid mailing list email
addresses are.

The list of email addresses changes over time ... so whomever creates a
new mailbox would need to remember to add it on the MX servers as well ...

Depending on what the exact setup is, a friend has a script that should
help: http://slacker.com/~nugget/projects/postfixrelaymaps/

In a nutshell, it pulls from things like /etc/passwd on the master MX
and then pushes that info out to the slaves. It's written in perl, so it
should be easy to modify to pull from whatever source is necessary.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Tue, 30 May 2006, Jim C. Nasby wrote:

Depending on what the exact setup is, a friend has a script that should
help: http://slacker.com/~nugget/projects/postfixrelaymaps/

Thanks, but the script would involve a fair amount of work, since our mail
system isn't based on the pasword file :) But, I have setup a cron job
that runs every 30 minutes to generate a relay_users map for the MX server
that contains all valid mailboxes on the system ... if someone does notice
an email bouncing to somewhere that *should* work, please do let me know
...

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy@hub.org MSN . scrappy@hub.org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664