List response time...

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups
scale a little better, but Usenet propagation delay was a problem even when
full feeds were below 100MB per day. Actually, Usenet propagation is better
now than then, now that the majority of sites aren't uucp and fed batched
with C-News. (or BNews, even....). Usenet propagation delays used to be
measured in days and sometimes weeks. To get a message in two days was great
time!

But I can remember when Usenet propagation delays were how you judged EXPIRE
times and newsspool size. And I also remember nasty tricks used when servers
that didn't respect 'distribution:' were hit with 'expires:' headers with
values below the mean propagation delay.....and I can recall getting CANCELS
for postings two days before the posting to be canceled came trickling in....

We're still not as bad as BugTraq, though. Not only is the message delayed
two to three days, other people will have already replied to it, and
discussion will have been closed off before I ever get a chance to say
anything. Well, maybe that's a good thing. :-)

I guess this IS one of the few advantages of having to use reply-all on this
list.... Although then the discussion has moved on before the general list
membership has had a chance to read most of the recent replies....

The best thing to do is simply to expect propagation delay.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar.owen@wgcr.org> writes:

The best thing to do is simply to expect propagation delay.

Actually, I just sent a gripe off to Marc about this. I've been
noticing large and variable propagation delay for a few months now,
but I just today realized that the problem is entirely local to hub.org.
For example, look at the headers on your message:

Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f7LJKpY10196
for <tgl@sss.pgh.pa.us>; Tue, 21 Aug 2001 15:20:51 -0400 (EDT)
Received: from postgresql.org.org (webmail.postgresql.org [216.126.85.28])
by postgresql.org (8.11.3/8.11.4) with SMTP id f7LJKpP46374;
Tue, 21 Aug 2001 15:20:52 -0400 (EDT)
(envelope-from pgsql-hackers-owner+M12441@postgresql.org)
Received: from www.wgcr.org (www.wgcr.org [206.74.232.194])
by postgresql.org (8.11.3/8.11.4) with ESMTP id f7LHxnP15711
for <pgsql-hackers@postgresql.org>; Tue, 21 Aug 2001 13:59:49 -0400 (EDT)
(envelope-from lamar.owen@wgcr.org)
Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3])
by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id NAA25357;
Tue, 21 Aug 2001 13:59:40 -0400

All the delay seems to be in transferring the message from
postgresql.org to webmail.postgresql.org ... which are the same
machine, or at least the same IP address. What's up with that?

regards, tom lane

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

Ian

Tom Lane <tgl@sss.pgh.pa.us> writes:

All the delay seems to be in transferring the message from
postgresql.org to webmail.postgresql.org ... which are the same
machine, or at least the same IP address. What's up with that?

You are seeing sendmail's poorly designed queuing behaviour in action.
sendmail limits itself by outgoing messages, rather than outgoing
deliveries. This causes one slow delivery to hold up many fast
deliveries.

Ian

I've had great luck with Postfix as well.

-Mitch

----- Original Message -----
From: "Ian Lance Taylor" <ian@airs.com>
To: "Lamar Owen" <lamar.owen@wgcr.org>
Cc: "Serguei Mokhov" <sa_mokho@alcor.concordia.ca>; "PostgreSQL Hackers"
<pgsql-hackers@postgresql.org>
Sent: Tuesday, August 21, 2001 4:24 PM
Subject: [HACKERS] Re: List response time...

On 21 Aug 2001, Ian Lance Taylor wrote:

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

ooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhh.... I've been raggin on
Marc on that one for well over a year, maybe two.. I started using
qmail when it was still in .7something beta and never looked back. The
folks at Security Focus have moved all of the lists to ezmlm (part of
qmail) and have had nothing but success... But don't tell Marc.

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Huh? Two different machines altogether ... but, I do have work to do once
the new server goes online ...

nslookup postgresql.org

Server: localhost.hub.org
Address: 127.0.0.1

Name: postgresql.org
Address: 216.126.84.28

nslookup webmail.postgresql.org

Server: localhost.hub.org
Address: 127.0.0.1

Name: mail.postgresql.org
Address: 216.126.85.28
Aliases: webmail.postgresql.org

On Tue, 21 Aug 2001, Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

Huh? Two different machines altogether ...

Hmm. Maybe the problem is this:

$ nslookup -q=mx postgresql.org
Server: localhost
Address: 127.0.0.1

Non-authoritative answer:
postgresql.org preference = 0, mail exchanger = mail.postgresql.org
postgresql.org preference = 20, mail exchanger = mail1.hub.org
postgresql.org preference = 20, mail exchanger = mailserv.hub.org

Authoritative answers can be found from:
postgresql.org nameserver = NS.TRENDS.CA
postgresql.org nameserver = NS.hub.org

mail.postgresql.org internet address = 216.126.85.28
mail1.hub.org internet address = 216.126.85.1
mailserv.hub.org internet address = 216.126.84.253
NS.TRENDS.CA internet address = 209.47.148.2
NS.hub.org internet address = 216.126.84.1

From out here, the primary mail acceptor for 'postgresql.org' shows
as mail.postgresql.org = 216.126.85.28. Should we be preferring
216.126.84.28 instead? It sure looks like there's an unnecessary
hop happening inside hub.org.

regards, tom lane

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Can I put qmail on it first?

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

Nope, but thanks for the offer ;)

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

Nope, but thanks for the offer ;)

Pleeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeze?????

You won't be sorry or disappointed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Can I put qmail on it first?

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

Ian Lance Taylor <ian@airs.com> writes:

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

The MTA used for various redhat.com mailing lists is postfix (and
mailman as listmanager)

--
Trond Eivind Glomsr�d
Red Hat, Inc.

If it was a sendmail issue, by all means, but it isn't so no :)

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

If it was a sendmail issue, by all means, but it isn't so no :)

Both qmail and postfix radically outperform sendmail for large mailing
list delivery on identical hardware. It seems strange to me to say
that there is no sendmail issue when sendmail itself is the issue.
The queuing structure sendmail uses is simply wrong when a single
message has many recipients. I've run moderately serious (1000 users,
dozens of messages per day) mailing lists using both sendmail and
qmail, and there really is no comparison.

Ian

Marc wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

Is there a quick(er) way to 'subscribe, set nomail' on all the mailing lists
that are mirrored to news.postgresql.org?

I prefer to read/post through the news server and I've had to subscribe
manually to most lists.

Cheers,

Colin

On Tuesday 21 August 2001 19:45, Marc G. Fournier wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane) addressed
to only the hackers list (which I know he's subscribed to). Up to a week
after reading it once already.

My mail spool filesystem has severl GB of free space, too. Unless my
sendmail installation is doing funny things. :-)
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar.owen@wgcr.org> writes:

On Tuesday 21 August 2001 19:45, Marc G. Fournier wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane) addressed
to only the hackers list (which I know he's subscribed to).

The approval delay isn't only for people who are not subscribed. Marc
also has a number of filters in there that are intended to shunt off
administrivia (ie, people who send uns*bscribe commands to the whole
list). This is not a bad idea, but unfortunately, his filter patterns
are WAY too loose IMHO. I've had posts delayed because of references
to c*ncel, s*b-SELECT, and some other words that I could hardly even
see the connection to administrivia requests.

Hoping that this post gets through without being delayed ;-)

regards, tom lane

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane)

addressed

to only the hackers list (which I know he's subscribed to). Up to a week
after reading it once already.

I can confirm this also. I have seen delayed (up to several days later)
duplicates of emails I have already received.

David Ford wrote:

Ian Lance Taylor wrote:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

It's all in the configuration. I slam mails around dozens of machines
in seconds using sendmail and I process a lot of mail.

Not only configuration. A friend of mine upgraded a computer that was
unable
to handle the mail feed from P200 to PIII 800 going from sendmail to
qmail at
the same time. The load average dropped from "allways very busy" to
0.02.

It is possible that it is mainly from better conf and faster processor
but then
I'd claim that qmail is easier to configure for big load.

----------------
Hannu

All the delay seems to be in transferring the message from
postgresql.org to webmail.postgresql.org ... which are the same
machine, or at least the same IP address. What's up with that?

Looks like sendmail? Change your queue runs to be more aggressive. I
have an mc file on http://blue-labs.org/clue/bluelabs.mc that has some
aggressive queue definitions.

David

Ian Lance Taylor wrote:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

It's all in the configuration. I slam mails around dozens of machines
in seconds using sendmail and I process a lot of mail.

David

You are seeing sendmail's poorly designed queuing behaviour in action.
sendmail limits itself by outgoing messages, rather than outgoing
deliveries. This causes one slow delivery to hold up many fast
deliveries.

Again, all in the configuration....rinse, repeat.

Simply change your queue priority.

David

Vince Vielhaber wrote:

ooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhh.... I've been raggin on
Marc on that one for well over a year, maybe two.. I started using
qmail when it was still in .7something beta and never looked back. The
folks at Security Focus have moved all of the lists to ezmlm (part of
qmail) and have had nothing but success... But don't tell Marc.

And ezlm is -ever- so quick to tell you your mail is bouncing when your
link goes down for a few hours or is sporadic. I know of several others
that simply send you the emails that are in queue.

-d

Ian Lance Taylor wrote:

Both qmail and postfix radically outperform sendmail for large mailing
list delivery on identical hardware. It seems strange to me to say
that there is no sendmail issue when sendmail itself is the issue.
The queuing structure sendmail uses is simply wrong when a single
message has many recipients. I've run moderately serious (1000 users,
dozens of messages per day) mailing lists using both sendmail and
qmail, and there really is no comparison.

Ian, please

It's in the configuration. I run much more than the above and have no
issues at all.

-d

It's in the configuration. I run much more than the above and have no
issues at all.

Yeah, some people shouldn't have root even if they own the machine.

On Fri, 24 Aug 2001, David Ford wrote:

It's all in the configuration. I slam mails around dozens of machines
in seconds using sendmail and I process a lot of mail.

So have you patched for the latest of the many sendmail root exploits?

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

Vince Vielhaber wrote:

On Fri, 24 Aug 2001, David Ford wrote:

It's all in the configuration. I slam mails around dozens of machines
in seconds using sendmail and I process a lot of mail.

So have you patched for the latest of the many sendmail root exploits?

Vince.

I keep my systems up to latest and greatest that passes the lab.
Currently 8.12.0b19. Since I keep things up to date and read the
documentation... I tend to avoid most security problems. Do keep in
mind that most of the latest issues are symbiotic problems due to issues
found in LK capabilities.

-d

speedboy <speedboy@nomicrosoft.org> writes:

It's in the configuration. I run much more than the above and have no
issues at all.

Yeah, some people shouldn't have root even if they own the machine.

Since I was the original poster I'm going to take minor umbrage. I've
been writing and distributing free software for over ten years, and my
work can be found in every Linux and *BSD distribution. What have you
done for the world lately?

I also do know how to configure sendmail, another thing I did for over
ten years until I switched to qmail in 1998. I will have to
respectfully disagree with David Ford, with the proviso that it is
certainly possible that recent sendmail releases have better queuing
behaviour. David, have you ever tried qmail or postfix? Why not?

Ian

David Ford <david@blue-labs.org> writes:

ooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhh.... I've been raggin on
Marc on that one for well over a year, maybe two.. I started using
qmail when it was still in .7something beta and never looked back. The
folks at Security Focus have moved all of the lists to ezmlm (part of
qmail) and have had nothing but success... But don't tell Marc.

And ezlm is -ever- so quick to tell you your mail is bouncing when
your link goes down for a few hours or is sporadic. I know of several
others that simply send you the emails that are in queue.

I don't know what you are referring to here. ezmlm simply handles
bounces generated by the MTA. qmail does not bounce mail merely
because a link goes down for a few hours or is sporadic.

There is an issue here which you may be referring to: vanilla ezmlm
does not handle temporary failure DSN notices very well--it treats
them as bounces. This is easily fixable, and in fact I believe that
ezmlm+idx (which is what most people use) does handle them correctly
by default.

Ian

Hey guys,

Can you move this thread elsewhere?

It's EXTREMELY off topic now.

:(

Regards and best wishes,

Justin Clift

Ian Lance Taylor wrote:

David Ford <david@blue-labs.org> writes:

ooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhh.... I've been raggin on
Marc on that one for well over a year, maybe two.. I started using
qmail when it was still in .7something beta and never looked back. The
folks at Security Focus have moved all of the lists to ezmlm (part of
qmail) and have had nothing but success... But don't tell Marc.

And ezlm is -ever- so quick to tell you your mail is bouncing when
your link goes down for a few hours or is sporadic. I know of several
others that simply send you the emails that are in queue.

I don't know what you are referring to here. ezmlm simply handles
bounces generated by the MTA. qmail does not bounce mail merely
because a link goes down for a few hours or is sporadic.

There is an issue here which you may be referring to: vanilla ezmlm
does not handle temporary failure DSN notices very well--it treats
them as bounces. This is easily fixable, and in fact I believe that
ezmlm+idx (which is what most people use) does handle them correctly
by default.

Ian

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi