Can we include capturing logs of pgdata/pg_upgrade_output.d/*/log in buildfarm
Hi,
Recently there have been few upgrade tap test failures in buildfarm
like in [1]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10 & [2]https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20. Analysing these failures requires the log files
that are getting generated from src/bin/pg_upgrade at the following
locations:
tmp_check/*/pgdata/pg_upgrade_output.d/*/*.txt - e.g.
tmp_check/t_004_subscription_new_sub1_data/pgdata/pg_upgrade_output.d/20240214T052229.045/subs_invalid.txt
tmp_check/*/pgdata/pg_upgrade_output.d/*/*/*.log - e.g.
tmp_check/t_004_subscription_new_sub1_data/pgdata/pg_upgrade_output.d/20240214T052229.045/log/pg_upgrade_server.log
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.
Can we include these log files also in the buildfarm?
[1]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2]: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
Regards,
Vignesh
On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.Can we include these log files also in the buildfarm?
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
Indeed, these lack some patterns. Why not sending a pull request
around [1]https://github.com/PGBuildFarm/client-code/blob/main/PGBuild/Modules/TestUpgrade.pm -- Michael to get more patterns covered?
[1]: https://github.com/PGBuildFarm/client-code/blob/main/PGBuild/Modules/TestUpgrade.pm -- Michael
--
Michael
On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.Can we include these log files also in the buildfarm?
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20Indeed, these lack some patterns. Why not sending a pull request
around [1] to get more patterns covered?
I have added a few more patterns to include the pg_upgrade generated
files. The attached patch has the changes for the same.
Adding Andrew also to get his thoughts on this.
Regards,
Vignesh
Attachments:
Include_pg_upgrade_generated_logs.patchtext/x-patch; charset=US-ASCII; name=Include_pg_upgrade_generated_logs.patchDownload
diff --git a/PGBuild/Modules/TestUpgrade.pm b/PGBuild/Modules/TestUpgrade.pm
index ad3e00e..83f62b7 100644
--- a/PGBuild/Modules/TestUpgrade.pm
+++ b/PGBuild/Modules/TestUpgrade.pm
@@ -139,6 +139,8 @@ sub check
$self->{pgsql}/src/bin/pg_upgrade/log/*
$self->{pgsql}/src/bin/pg_upgrade/tmp_check/*/*.diffs
$self->{pgsql}/src/bin/pg_upgrade/tmp_check/data/pg_upgrade_output.d/log/*
+ $self->{pgsql}/src/bin/pg_upgrade/tmp_check/*/pgdata/pg_upgrade_output.d/*/*.txt
+ $self->{pgsql}/src/bin/pg_upgrade/tmp_check/*/pgdata/pg_upgrade_output.d/*/log/*.log
$self->{pgsql}/src/test/regress/*.diffs"
);
$log->add_log($_) foreach (@logfiles);
On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.Can we include these log files also in the buildfarm?
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20Indeed, these lack some patterns. Why not sending a pull request
around [1] to get more patterns covered?I have added a few more patterns to include the pg_upgrade generated
files. The attached patch has the changes for the same.
Adding Andrew also to get his thoughts on this.
I have added the following commitfest entry for this:
https://commitfest.postgresql.org/47/4850/
Regards,
Vignesh
On 2024-02-25 Su 11:18, vignesh C wrote:
On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.Can we include these log files also in the buildfarm?
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20Indeed, these lack some patterns. Why not sending a pull request
around [1] to get more patterns covered?I have added a few more patterns to include the pg_upgrade generated
files. The attached patch has the changes for the same.
Adding Andrew also to get his thoughts on this.I have added the following commitfest entry for this:
https://commitfest.postgresql.org/47/4850/
Buildfarm code patches do not belong in the Commitfest, I have marked
the item as rejected. You can send me patches directly or add a PR to
the buildfarm's github repo.
In this case the issue on drongo was a typo, the fix for which I had
forgotten to propagate back in December. Note that the buildfarm's
TestUpgrade.pm module is only used for branches < 15. For branches >= 15
we run the standard TAP test and this module does nothing.
More generally, the collection of logs etc. for pg_upgrade will improve
with the next release, which will be soon after I return from a vacation
in about 2 weeks - experience shows that making releases just before a
vacation is not a good idea :-)
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Mon, 26 Feb 2024 at 10:57, Andrew Dunstan <andrew@dunslane.net> wrote:
On 2024-02-25 Su 11:18, vignesh C wrote:
On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.Can we include these log files also in the buildfarm?
[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20Indeed, these lack some patterns. Why not sending a pull request
around [1] to get more patterns covered?I have added a few more patterns to include the pg_upgrade generated
files. The attached patch has the changes for the same.
Adding Andrew also to get his thoughts on this.I have added the following commitfest entry for this:
https://commitfest.postgresql.org/47/4850/Buildfarm code patches do not belong in the Commitfest, I have marked
the item as rejected. You can send me patches directly or add a PR to
the buildfarm's github repo.
Ok, I will send over the patch directly for the required things.
In this case the issue on drongo was a typo, the fix for which I had
forgotten to propagate back in December. Note that the buildfarm's
TestUpgrade.pm module is only used for branches < 15. For branches >= 15
we run the standard TAP test and this module does nothing.More generally, the collection of logs etc. for pg_upgrade will improve
with the next release, which will be soon after I return from a vacation
in about 2 weeks - experience shows that making releases just before a
vacation is not a good idea :-)
Thanks, that will be helpful.
Regards,
Vignesh