List response time...

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups
scale a little better, but Usenet propagation delay was a problem even when
full feeds were below 100MB per day. Actually, Usenet propagation is better
now than then, now that the majority of sites aren't uucp and fed batched
with C-News. (or BNews, even....). Usenet propagation delays used to be
measured in days and sometimes weeks. To get a message in two days was great
time!

But I can remember when Usenet propagation delays were how you judged EXPIRE
times and newsspool size. And I also remember nasty tricks used when servers
that didn't respect 'distribution:' were hit with 'expires:' headers with
values below the mean propagation delay.....and I can recall getting CANCELS
for postings two days before the posting to be canceled came trickling in....

We're still not as bad as BugTraq, though. Not only is the message delayed
two to three days, other people will have already replied to it, and
discussion will have been closed off before I ever get a chance to say
anything. Well, maybe that's a good thing. :-)

I guess this IS one of the few advantages of having to use reply-all on this
list.... Although then the discussion has moved on before the general list
membership has had a chance to read most of the recent replies....

The best thing to do is simply to expect propagation delay.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar.owen@wgcr.org> writes:

The best thing to do is simply to expect propagation delay.

Actually, I just sent a gripe off to Marc about this. I've been
noticing large and variable propagation delay for a few months now,
but I just today realized that the problem is entirely local to hub.org.
For example, look at the headers on your message:

Received: from postgresql.org (webmail.postgresql.org [216.126.85.28])
by sss.pgh.pa.us (8.11.4/8.11.4) with ESMTP id f7LJKpY10196
for <tgl@sss.pgh.pa.us>; Tue, 21 Aug 2001 15:20:51 -0400 (EDT)
Received: from postgresql.org.org (webmail.postgresql.org [216.126.85.28])
by postgresql.org (8.11.3/8.11.4) with SMTP id f7LJKpP46374;
Tue, 21 Aug 2001 15:20:52 -0400 (EDT)
(envelope-from pgsql-hackers-owner+M12441@postgresql.org)
Received: from www.wgcr.org (www.wgcr.org [206.74.232.194])
by postgresql.org (8.11.3/8.11.4) with ESMTP id f7LHxnP15711
for <pgsql-hackers@postgresql.org>; Tue, 21 Aug 2001 13:59:49 -0400 (EDT)
(envelope-from lamar.owen@wgcr.org)
Received: from lowen.wgcr.org (IDENT:lowen@[10.1.2.3])
by www.wgcr.org (8.9.3/8.9.3/WGCR) with SMTP id NAA25357;
Tue, 21 Aug 2001 13:59:40 -0400

All the delay seems to be in transferring the message from
postgresql.org to webmail.postgresql.org ... which are the same
machine, or at least the same IP address. What's up with that?

regards, tom lane

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

Ian

Tom Lane <tgl@sss.pgh.pa.us> writes:

All the delay seems to be in transferring the message from
postgresql.org to webmail.postgresql.org ... which are the same
machine, or at least the same IP address. What's up with that?

You are seeing sendmail's poorly designed queuing behaviour in action.
sendmail limits itself by outgoing messages, rather than outgoing
deliveries. This causes one slow delivery to hold up many fast
deliveries.

Ian

I've had great luck with Postfix as well.

-Mitch

----- Original Message -----
From: "Ian Lance Taylor" <ian@airs.com>
To: "Lamar Owen" <lamar.owen@wgcr.org>
Cc: "Serguei Mokhov" <sa_mokho@alcor.concordia.ca>; "PostgreSQL Hackers"
<pgsql-hackers@postgresql.org>
Sent: Tuesday, August 21, 2001 4:24 PM
Subject: [HACKERS] Re: List response time...

On 21 Aug 2001, Ian Lance Taylor wrote:

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

ooooooooooooooooooooooohhhhhhhhhhhhhhhhhhhhhh.... I've been raggin on
Marc on that one for well over a year, maybe two.. I started using
qmail when it was still in .7something beta and never looked back. The
folks at Security Focus have moved all of the lists to ezmlm (part of
qmail) and have had nothing but success... But don't tell Marc.

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Huh? Two different machines altogether ... but, I do have work to do once
the new server goes online ...

nslookup postgresql.org

Server: localhost.hub.org
Address: 127.0.0.1

Name: postgresql.org
Address: 216.126.84.28

nslookup webmail.postgresql.org

Server: localhost.hub.org
Address: 127.0.0.1

Name: mail.postgresql.org
Address: 216.126.85.28
Aliases: webmail.postgresql.org

On Tue, 21 Aug 2001, Tom Lane wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

Huh? Two different machines altogether ...

Hmm. Maybe the problem is this:

$ nslookup -q=mx postgresql.org
Server: localhost
Address: 127.0.0.1

Non-authoritative answer:
postgresql.org preference = 0, mail exchanger = mail.postgresql.org
postgresql.org preference = 20, mail exchanger = mail1.hub.org
postgresql.org preference = 20, mail exchanger = mailserv.hub.org

Authoritative answers can be found from:
postgresql.org nameserver = NS.TRENDS.CA
postgresql.org nameserver = NS.hub.org

mail.postgresql.org internet address = 216.126.85.28
mail1.hub.org internet address = 216.126.85.1
mailserv.hub.org internet address = 216.126.84.253
NS.TRENDS.CA internet address = 209.47.148.2
NS.hub.org internet address = 216.126.84.1

From out here, the primary mail acceptor for 'postgresql.org' shows
as mail.postgresql.org = 216.126.85.28. Should we be preferring
216.126.84.28 instead? It sure looks like there's an unnecessary
hop happening inside hub.org.

regards, tom lane

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Can I put qmail on it first?

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

Nope, but thanks for the offer ;)

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

Nope, but thanks for the offer ;)

Pleeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeze?????

You won't be sorry or disappointed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

On Tue, 21 Aug 2001, Marc G. Fournier wrote:

On Tue, 21 Aug 2001, Lamar Owen wrote:

On Tuesday 21 August 2001 12:59, Serguei Mokhov wrote:

Looking at my message about the bug webpage and
some other posts, I see that it was delayed for
about 2h and a half. Some of the post were
delayed for days... Why is that? Looks like
the list has problems of some sort which cause
these irregular delays.

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)). Newsgroups

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

.. as far as server load is concerned, causing several hour delay, we're
just about to put online a dual processor server that has been donated to
the project ... its first task is going to be mailing list distribution
and a open news server for the newsgroups.

Can I put qmail on it first?

Vince.
--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

--
==========================================================================
Vince Vielhaber -- KA8CSH email: vev@michvhf.com http://www.pop4.net
56K Nationwide Dialup from $16.00/mo at Pop4 Networking
Online Campground Directory http://www.camping-usa.com
Online Giftshop Superstore http://www.cloudninegifts.com
==========================================================================

Ian Lance Taylor <ian@airs.com> writes:

Lamar Owen <lamar.owen@wgcr.org> writes:

Mailing lists don't scale well to large numbers of subscribers. I see this
delay constantly,on multiple lists. The bigger the list gets, the slower the
list gets (and the more loaded the server gets, right Marc? :-)).

Note that the postgresql.org mail server is still running sendmail.
In my personal experience with sources.redhat.com, qmail is a much
better choice to handle large mailing lists. When we switched from
sendmail to qmail, mailing list delays dropped from hours, or
sometimes even days, to seconds.

The MTA used for various redhat.com mailing lists is postfix (and
mailman as listmanager)

--
Trond Eivind Glomsr�d
Red Hat, Inc.

If it was a sendmail issue, by all means, but it isn't so no :)

On Tue, 21 Aug 2001, Vince Vielhaber wrote:

"Marc G. Fournier" <scrappy@hub.org> writes:

If it was a sendmail issue, by all means, but it isn't so no :)

Both qmail and postfix radically outperform sendmail for large mailing
list delivery on identical hardware. It seems strange to me to say
that there is no sendmail issue when sendmail itself is the issue.
The queuing structure sendmail uses is simply wrong when a single
message has many recipients. I've run moderately serious (1000 users,
dozens of messages per day) mailing lists using both sendmail and
qmail, and there really is no comparison.

Ian

Marc wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

Is there a quick(er) way to 'subscribe, set nomail' on all the mailing lists
that are mirrored to news.postgresql.org?

I prefer to read/post through the news server and I've had to subscribe
manually to most lists.

Cheers,

Colin

On Tuesday 21 August 2001 19:45, Marc G. Fournier wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane) addressed
to only the hackers list (which I know he's subscribed to). Up to a week
after reading it once already.

My mail spool filesystem has severl GB of free space, too. Unless my
sendmail installation is doing funny things. :-)
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar.owen@wgcr.org> writes:

On Tuesday 21 August 2001 19:45, Marc G. Fournier wrote:

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane) addressed
to only the hackers list (which I know he's subscribed to).

The approval delay isn't only for people who are not subscribed. Marc
also has a number of filters in there that are intended to shunt off
administrivia (ie, people who send uns*bscribe commands to the whole
list). This is not a bad idea, but unfortunately, his filter patterns
are WAY too loose IMHO. I've had posts delayed because of references
to c*ncel, s*b-SELECT, and some other words that I could hardly even
see the connection to administrivia requests.

Hoping that this post gets through without being delayed ;-)

regards, tom lane

Actually, the 'multi-day' delay is generally related to posts from ppl
that aren't subscribed to the lists that I have to approve manually ...

I have been getting delayed duplicates from people (ie, Tom Lane)

addressed

to only the hackers list (which I know he's subscribed to). Up to a week
after reading it once already.

I can confirm this also. I have seen delayed (up to several days later)
duplicates of emails I have already received.