reduce size of logs stored by buildfarm
Hi,
I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output. This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless. We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.
We also have a couple of tests that leave rather large server logs
behind that are perhaps not very useful. I would propose to run those
servers with log_statement=off and avoid some of the bloat. 0002 does
that.
Another possible approach, instead of individually tweaking t/*.pl files
to add 'log_statements=none', would be to set it that way in
PostgreSQL::Test::Cluster globally. Right now we have
log_statements=all there, but maybe that was not a great idea.
I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'. I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way. The
problem here is that we print 'sending JSON response' and the verbatim
response, included the 1 MB of "x" as '_pad_'. Can we supress this?
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB. Maybe there are better
ways to deal with this? Jacob?
With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB. I think that's not a bad reduction.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Are you not unsure you want to delete Firefox?
[Not unsure] [Not not unsure] [Cancel]
http://smylers.hates-software.com/2008/01/03/566e45b2.html
Hi,
On 2025-11-25 14:30:33 +0100, �lvaro Herrera wrote:
I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output. This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless. We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.
Yea, that makes no sense.
We also have a couple of tests that leave rather large server logs
behind that are perhaps not very useful. I would propose to run those
servers with log_statement=off and avoid some of the bloat. 0002 does
that.Another possible approach, instead of individually tweaking t/*.pl files
to add 'log_statements=none', would be to set it that way in
PostgreSQL::Test::Cluster globally. Right now we have
log_statements=all there, but maybe that was not a great idea.
I think that be a bad idea - many tests are essentially undebuggable that way
because there's no other way to correlate the regress_log and the server log
than the statement log.
TBH, I have even needed the pg_dump log_statement output on the buildfarm to
debug issues :(
I am against doing this for 027_stream_regress.pl, it's definitely
undebuggable without the statement logs.
Greetings,
Andres Freund
=?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:
With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB. I think that's not a bad reduction.
I wonder how much this overlaps with what Andrew just did
to the BF client [1]/messages/by-id/601bb91d-b55d-4fa6-bb57-c2126fb22620@dunslane.net.
regards, tom lane
[1]: /messages/by-id/601bb91d-b55d-4fa6-bb57-c2126fb22620@dunslane.net
On 2025-11-25 Tu 10:37 AM, Tom Lane wrote:
=?utf-8?Q?=C3=81lvaro?= Herrera <alvherre@kurilemu.de> writes:
With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB. I think that's not a bad reduction.I wonder how much this overlaps with what Andrew just did
to the BF client [1].[1] /messages/by-id/601bb91d-b55d-4fa6-bb57-c2126fb22620@dunslane.net
Pretty much none of it, although both Alvaro and I are clearly motivated
by the same thing, namely disk space pressure on the buildfarm server.
I think his patches 1 and 3 seem like no-brainers.
I did two things in the BF client:
. prevent uploading redundant copies of the same file in meson builds
(this was a bug)
. inhibit loading postmaster log files from installcheck runs and
cross-version upgrade runs, but only if there hasn't been a failure.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On 2025-Nov-25, Andrew Dunstan wrote:
I think his patches 1 and 3 seem like no-brainers.
On 2025-Nov-25, Andres Freund wrote:
On 2025-11-25 14:30:33 +0100, Álvaro Herrera wrote:
I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output. This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless. We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.Yea, that makes no sense.
Thanks, pushed 0001.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Postgres is bloatware by design: it was built to house
PhD theses." (Joey Hellerstein, SIGMOD annual conference 2002)
Import Notes
Reply to msg id not found: a9db7d10-a941-46bc-80b7-d9c616097af3@dunslane.netozycnnem6xbrsa3nhylpqy2kyl2kudq3r4sq2obxh6cm6evqsm@ozihkcefz3e4 | Resolved by subject fallback
On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'. I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way.
Sorry about that.
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB. Maybe there are better
ways to deal with this? Jacob?
Only thing I don't like about this is that the JSON you need for
debugging might be after the useless padding. Attached is a patch that
does things more surgically, tested against Python 3.6, and I'm
running it through the CI now.
Thanks,
--Jacob
Attachments:
truncate-JSON-logs.patch.txttext/plain; charset=UTF-8; name=truncate-JSON-logs.patch.txtDownload+21-2
On 2025-Nov-25, Jacob Champion wrote:
On Tue, Nov 25, 2025 at 5:30 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'. I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way.Sorry about that.
No need!
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB. Maybe there are better
ways to deal with this? Jacob?Only thing I don't like about this is that the JSON you need for
debugging might be after the useless padding. Attached is a patch that
does things more surgically, tested against Python 3.6, and I'm
running it through the CI now.
Brilliant, thanks.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"You're _really_ hosed if the person doing the hiring doesn't understand
relational systems: you end up with a whole raft of programmers, none of
whom has had a Date with the clue stick." (Andrew Sullivan)
/messages/by-id/20050809113420.GD2768@phlogiston.dyndns.org
On Tue, Nov 25, 2025 at 10:46 AM Álvaro Herrera <alvherre@kurilemu.de> wrote:
Only thing I don't like about this is that the JSON you need for
debugging might be after the useless padding. Attached is a patch that
does things more surgically, tested against Python 3.6, and I'm
running it through the CI now.Brilliant, thanks.
(This was pushed last week; forgot to follow up here.)
Thanks,
--Jacob