Postgres mail list traffic over time

Tom Lane wrote:

I got interested by Bruce's plot of PG email traffic here
http://momjian.us/main/img/pgincoming.gif
and decided to try to extend it into the past. The data I have
available is just my own incoming mail log, but being a pack-rat by
nature I have that back to April 1998. Attached is a graph of Postgres
list messages per month since then. I should note that this covers only
the mail lists I'm subscribed to, which has been most of them since
about 1999; but the first few numbers in this chart are undercounts by
comparison. Also, the very last dot is month-to-date for November and
so is an underestimate.

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

Yes, I know Magnus did a graph for the PG-EU conference and it was also
flat; perhaps he can post it here. His chart was pulled from the
Postgres archives, so it is even more accurate than our graphs.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related. I am going to post
another blog tomorrow with more thoughts.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Does anyone have the data to break it down per mailing list? That might
yield some more insight.

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Agreed. I am struggling to put into words some of my angst, but I am
concerned I will not be able to offer the same guarantees I have done in
previous releases that every bug has been either fixed or added to the
TODO list, and every submitted patch has been either applied or rejected.

There, I said it. :-(

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Thu, 2008-11-20 at 22:36 -0500, Tom Lane wrote:

I got interested by Bruce's plot of PG email traffic here
http://momjian.us/main/img/pgincoming.gifto
and decided to try to extend it into the past. The data I have
available is just my own incoming mail log, but being a pack-rat by
nature I have that back to April 1998. Attached is a graph of Postgres
list messages per month since then. I should note that this covers only
the mail lists I'm subscribed to, which has been most of them since
about 1999; but the first few numbers in this chart are undercounts by
comparison. Also, the very last dot is month-to-date for November and
so is an underestimate.

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

Am I reading your graph wrong? I show a sharp increase right before 2006
and then a small drop off but a constant after that?

I know that my email (I am pretty sure I am subscribed to at least as
many lists as you) has been on a steady incline, especially through
-general and -hackers.

Joshua D. Drake

regards, tom lane

--

"Joshua D. Drake" <jd@commandprompt.com> writes:

I know that my email (I am pretty sure I am subscribed to at least as
many lists as you) has been on a steady incline, especially through
-general and -hackers.

I would have said the same, which is why I find it noteworthy that
my mail logs don't seem to support that impression. Have you got
actual log data on the point?

regards, tom lane

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Perhaps it's just subjective: we're all getting older.

Soon, these pesky whippersnappers will want to twitter their PG
questions to this list over YouTube.

On Thu, 2008-11-20 at 23:46 -0500, Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

I know that my email (I am pretty sure I am subscribed to at least as
many lists as you) has been on a steady incline, especially through
-general and -hackers.

I would have said the same, which is why I find it noteworthy that
my mail logs don't seem to support that impression. Have you got
actual log data on the point?

I purge my postgresql logs except for some specific ones (like PGFG).
however, I have the entire archives.postgresql.org.

pgsql-hackers (since inception, 1997), first line date, second line
number of messages.

1997-01
939
1997-02
300
1997-03
534
1997-04
865
1997-05
484
1997-06
601
1997-07
392
1997-08
399
1997-09
579
1997-10
594
1997-11
381
1997-12
351
1998-01
870
1998-02
1326
1998-03
1121
1998-04
707
1998-05
632
1998-06
493
1998-07
490
1998-08
867
1998-09
675
1998-10
1221
1998-11
609
1998-12
600
1999-01
769
1999-02
699
1999-03
1008
1999-04
217
1999-05
1155
1999-06
1241
1999-07
1052
1999-08
705
1999-09
945
1999-10
962
1999-11
929
1999-12
1065
2000-01
1688
2000-02
1460
2000-03
288
2000-04
187
2000-05
1686
2000-06
1283
2000-07
1477
2000-08
890
2000-09
642
2000-10
1320
2000-11
1419
2000-12
1234
2001-01
1469
2001-02
1178
2001-03
1708
2001-04
1181
2001-05
1478
2001-06
1151
2001-07
955
2001-08
1220
2001-09
921
2001-10
1165
2001-11
1318
2001-12
970
2002-01
1411
2002-02
1233
2002-03
1246
2002-04
1565
2002-05
1169
2002-06
1045
2002-07
1339
2002-08
2308
2002-09
1843
2002-10
1469
2002-11
1257
2002-12
1172
2003-01
1356
2003-02
1324
2003-03
1262
2003-04
1033
2003-05
812
2003-06
1316
2003-07
1068
2003-08
1373
2003-09
1695
2003-10
1631
2003-11
1643
2003-12
836
2004-01
878
2004-02
1017
2004-03
1352
2004-04
1177
2004-05
1495
2004-06
1025
2004-07
1430
2004-08
1620
2004-09
953
2004-10
1084
2004-11
1226
2004-12
963
2005-01
1116
2005-02
987
2005-03
1086
2005-04
1022
2005-05
1626
2005-06
1598
2005-07
1162
2005-08
1217
2005-09
1484
2005-10
1442
2005-11
1587
2005-12
1278
2006-01
1050
2006-02
1282
2006-03
1343
2006-04
1158
2006-05
1386
2006-06
1645
2006-07
1660
2006-08
2060
2006-09
2397
2006-10
1583
2006-11
1031
2006-12
1437
2007-01
1663
2007-02
1953
2007-03
1871
2007-04
1285
2007-05
1201
2007-06
1140
2007-07
1019
2007-08
1244
2007-09
1230
2007-10
1575
2007-11
1380
2007-12
1000
2008-01
1236
2008-02
1324
2008-03
1308
2008-04
1928
2008-05
1128
2008-06
1161
2008-07
1512
2008-08
1391
2008-09
1910
2008-10
1715
2008-11
1431

regards, tom lane

--

On Thu, 2008-11-20 at 21:19 -0800, Joshua D. Drake wrote:

On Thu, 2008-11-20 at 23:46 -0500, Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

I know that my email (I am pretty sure I am subscribed to at least as
many lists as you) has been on a steady incline, especially through
-general and -hackers.

I would have said the same, which is why I find it noteworthy that
my mail logs don't seem to support that impression. Have you got
actual log data on the point?

I purge my postgresql logs except for some specific ones (like PGFG).
however, I have the entire archives.postgresql.org.

pgsql-hackers (since inception, 1997), first line date, second line
number of messages.

pgsql-general

1998-05
139
1998-06
337
1998-07
438
1998-08
226
1998-09
187
1998-10
283
1998-11
269
1998-12
242
1999-01
302
1999-02
356
1999-03
385
1999-04
332
1999-05
404
1999-06
470
1999-07
411
1999-08
496
1999-09
385
1999-10
606
1999-11
512
1999-12
631
2000-01
667
2000-02
477
2000-03
219
2000-04
705
2000-05
843
2000-06
803
2000-07
1180
2000-08
861
2000-09
999
2000-10
1337
2000-11
1084
2000-12
1002
2001-01
1700
2001-02
1623
2001-03
1656
2001-04
1568
2001-05
1710
2001-06
1651
2001-07
1342
2001-08
1303
2001-09
1195
2001-10
1223
2001-11
1124
2001-12
901
2002-01
1216
2002-02
1419
2002-03
1388
2002-04
1287
2002-05
1192
2002-06
1366
2002-07
1893
2002-08
1261
2002-09
1438
2002-10
1444
2002-11
1517
2002-12
1225
2003-01
1657
2003-02
1760
2003-03
1597
2003-04
1611
2003-05
1295
2003-06
1951
2003-07
1586
2003-08
1836
2003-09
1880
2003-10
1604
2003-11
1768
2003-12
1664
2004-01
1708
2004-02
1355
2004-03
1215
2004-04
1210
2004-05
965
2004-06
1236
2004-07
973
2004-08
1677
2004-09
1337
2004-10
1579
2004-11
1557
2004-12
1358
2005-01
1877
2005-02
1535
2005-03
1622
2005-04
1460
2005-05
1379
2005-06
1413
2005-07
1332
2005-08
1632
2005-09
1232
2005-10
1945
2005-11
1438
2005-12
1402
2006-01
1743
2006-02
1218
2006-03
1602
2006-04
1372
2006-05
1604
2006-06
1268
2006-07
1170
2006-08
1501
2006-09
1289
2006-10
1588
2006-11
1866
2006-12
1619
2007-01
1953
2007-02
1720
2007-03
1724
2007-04
1304
2007-05
1650
2007-06
1796
2007-07
1257
2007-08
2097
2007-09
1385
2007-10
1722
2007-11
1770
2007-12
1487
2008-01
1621
2008-02
1527
2008-03
1666
2008-04
1446
2008-05
1144
2008-06
1055
2008-07
1251
2008-08
1188
2008-09
1252
2008-10
1485
2008-11
1045
--

On Fri, 2008-11-21 at 00:06 -0500, brian wrote:

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Perhaps it's just subjective: we're all getting older.

ouch

Soon, these pesky whippersnappers will want to twitter their PG
questions to this list over YouTube.

I assume you don't realize that is already happening :P

Joshua D. Drake

--

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

The numbers posted show a slow but steady increase, but I am wondering if there's more distinct subjects ?

Can we get a count on distinct threads per month (obviously some slop as some threads last for a while).

Greg Williamson
Senior DBA
DigitalGlobe

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and must be protected in accordance with those provisions. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

(My corporate masters made me say this.)

Magnus Hagander wrote:

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Does anyone have the data to break it down per mailing list? That might
yield some more insight.

Here's a graph of the more popular mailinglists (I couldn't include all
- the graph was completely unreadable) as seen in the archives search db.

Pfft, -general didn't like that file even though it was only 60k or so.

Here's a link to an uploaded version:
http://www.smugmug.com/photos/421507651_8pe6C-O.png

//Magnus

Tom Lane wrote:

"Joshua D. Drake" <jd@commandprompt.com> writes:

I know that my email (I am pretty sure I am subscribed to at least as
many lists as you) has been on a steady incline, especially through
-general and -hackers.

I would have said the same, which is why I find it noteworthy that
my mail logs don't seem to support that impression. Have you got
actual log data on the point?

Markmail shows some graphs. The one on the "main page" gives the
traffic for all the lists:
http://pgsql.markmail.org/

If you search for "pgsql-general" you get a graph for that list:
http://pgsql.markmail.org/search/?q=list%3Aorg.postgresql.pgsql-general

Same for -hackers:
http://pgsql.markmail.org/search/?q=list%3Aorg.postgresql.pgsql-hackers

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Thu, Nov 20, 2008 at 10:59:31PM -0500, Tom Lane wrote:

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

Does anyone have the data to break it down per mailing list? That might
yield some more insight.

The markmail archives generate pretty graphs and they seem to have a
good coverage from quite a few of the lists. e.g.:

http://markmail.org/search/?q=list:org.postgresql.pgsql-general
http://markmail.org/search/?q=list:org.postgresql.pgsql-hackers

the following has links to more:

http://markmail.org/search/?q=list:org.postgresql

be interesting to see how their servers take the hammering!

Sam

On Fri, 2008-11-21 at 10:43 -0300, Alvaro Herrera wrote:

Tom Lane wrote:

Markmail shows some graphs. The one on the "main page" gives the
traffic for all the lists:
http://pgsql.markmail.org/

If you search for "pgsql-general" you get a graph for that list:
http://pgsql.markmail.org/search/?q=list%3Aorg.postgresql.pgsql-general

Same for -hackers:
http://pgsql.markmail.org/search/?q=list%3Aorg.postgresql.pgsql-hackers

The top "Who sent it" list is very telling. It says, "Paging Tom Lane...
take a vacation!" :)

Joshua D. Drake

--

Sam Mason wrote:

the following has links to more:

http://markmail.org/search/?q=list:org.postgresql

Wow, the spanish list is the 3rd in traffic after hackers and general!

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Bruce Momjian wrote:

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

If measured in "bytes of the gzipped mbox" it looks like there's a
*huge* increase of volume on Hackers in the past 3 months - well
over twice the historical levels; and maybe 4X 2002-2006.

Graphs of this metric can be seen here:

http://0ape.com/postgres_mailinglist_size/

In some ways I think compressed mbox sizes are a more fair way
of measuring the bandwidth for these lists since it (correctly)
counts a large gzipped path as requiring more mental effort than
people top-posting brief messages on top of old threads.

(Data from commands like
HEAD http://archives.postgresql.org/pgsql-hackers/mbox/pgsql-hackers.2008-09.gz | grep Content-Length
)

On Thursday 20 November 2008 7:59:31 pm Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

Tom Lane wrote:

So, to a first approximation, the PG list traffic has been constant
since 2000. Not the result I expected.

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

I am still relatively new to Postgres, but my impression is that the questions
have gotten harder/more in depth. Fewer, How do you pronounce Postgres? and
more, Explain the various isolation levels for transactions and how does that
affect my particular situation?

Does anyone have the data to break it down per mailing list? That might
yield some more insight.

regards, tom lane

--
Adrian Klaver
aklaver@comcast.net

On Fri, 2008-11-21 at 08:18 -0800, Ron Mayer wrote:

Bruce Momjian wrote:

Tom Lane wrote:

Bruce Momjian <bruce@momjian.us> writes:

I also was confused by its flatness. I am finding the email traffic
almost impossible to continue tracking, so something different is
happening, but it seems it is not volume-related.

Yes, my perception also is that it's getting harder and harder to keep
up with the list traffic; so something is happening that a simple
volume count doesn't capture.

If measured in "bytes of the gzipped mbox" it looks like there's a
*huge* increase of volume on Hackers in the past 3 months - well
over twice the historical levels; and maybe 4X 2002-2006.

Its because we eliminated the -patches mailing list.

Joshua D. Drake

--

"Joshua D. Drake" <jd@commandprompt.com> writes:

On Fri, 2008-11-21 at 08:18 -0800, Ron Mayer wrote:

If measured in "bytes of the gzipped mbox" it looks like there's a
*huge* increase of volume on Hackers in the past 3 months - well
over twice the historical levels; and maybe 4X 2002-2006.

Its because we eliminated the -patches mailing list.

Yeah, I think this is most probably explained by repeat postings
of successive versions of large patches. Still, Ron might be on to
something. I had not considered message lengths in my previous
numbers ...

regards, tom lane