Allow pg_archivecleanup to remove backup history files
Hi,
Currently pg_archivecleanup doesn't remove backup history files even
when they're older than oldestkeptwalfile.
Of course the size of backup history files are smaller than WAL files
and they wouldn't consume much disk space, but a lot of backup history
files(e.g. daily backup for years) whose backups data have been already
removed are unnecessary and I would appreciate if pg_archivecleanup has
an option to remove them.
Attached a PoC patch, which added new option -b to remove files
including backup history files older than oldestkeptwalfile.
$ ls archivedir
000000010000000000000001 000000010000000000000003
000000010000000000000006
000000010000000000000008
000000010000000000000002 000000010000000000000004
000000010000000000000007
000000010000000000000009
000000010000000000000002.00000028.backup 000000010000000000000005
000000010000000000000007.00000028.backup
00000001000000000000000A.partial
$ pg_archivecleanup -b archivedir 000000010000000000000009
$ ls archivedir
000000010000000000000009 00000001000000000000000A.partial
Any thoughts?
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v1-0001-Allow-pg_archivecleanup-to-remove-backuphistoryfile.patchtext/x-diff; name=v1-0001-Allow-pg_archivecleanup-to-remove-backuphistoryfile.patchDownload
From ad87224ec5ba6ee13ccf934bf3e5adefb7e67212 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 24 Apr 2023 23:28:06 +0900
Subject: [PATCH v1] Allow pg_archivecleanup to remove backup history files
Add new option -b to remove files including backup history files older
than oldestkeptwalfile.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..5bc90fbadd 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool removeBackupHistoryFile = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -118,7 +120,8 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
+ if (((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) ||
+ (removeBackupHistoryFile && IsBackupHistoryFileName(walfile))) &&
strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
{
char WALFilePath[MAXPGPATH * 2]; /* the file path
@@ -252,6 +255,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b remove files including backup history files\n"));
printf(_(" -d generate debug output (verbose mode)\n"));
printf(_(" -n dry run, show the names of the files that would be removed\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -294,10 +298,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt(argc, argv, "bdnx:")) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ removeBackupHistoryFile = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
base-commit: c5e4ec293ea527394fc1d0006ab88047b3ce580f
--
2.25.1
At Tue, 25 Apr 2023 16:38:16 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
Hi,
Currently pg_archivecleanup doesn't remove backup history files even
when they're older than oldestkeptwalfile.Of course the size of backup history files are smaller than WAL files
and they wouldn't consume much disk space, but a lot of backup history
files(e.g. daily backup for years) whose backups data have been
already removed are unnecessary and I would appreciate if
pg_archivecleanup has an option to remove them.Attached a PoC patch, which added new option -b to remove files
including backup history files older than oldestkeptwalfile.$ ls archivedir
000000010000000000000001 000000010000000000000003
000000010000000000000006
000000010000000000000008
000000010000000000000002 000000010000000000000004
000000010000000000000007
000000010000000000000009
000000010000000000000002.00000028.backup 000000010000000000000005
000000010000000000000007.00000028.backup
00000001000000000000000A.partial$ pg_archivecleanup -b archivedir 000000010000000000000009
$ ls archivedir
000000010000000000000009 00000001000000000000000A.partialAny thoughts?
I thought that we have decided not to do that, but I coundn't find any
discussion about it in the ML archive. Anyway, I think it is great
that we have that option.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Tue, Apr 25, 2023 at 05:29:48PM +0900, Kyotaro Horiguchi wrote:
I thought that we have decided not to do that, but I coundn't find any
discussion about it in the ML archive. Anyway, I think it is great
that we have that option.
No objections from here to make that optional. It's been argued for a
couple of times that leaving the archive history files is good for
debugging, but I can also get why one would one to clean up all the
files older than a WAL retention policy. Even if these are just few
bytes, it can be noisy for the eye to see a large accumulation of
history files mixed with the WAL segments.
--
Michael
Horiguchi-san, Michael-san
Thanks for your comments and information!
Attached a patch with documentation and regression tests.
On 2023-04-26 06:39, Michael Paquier wrote:
On Tue, Apr 25, 2023 at 05:29:48PM +0900, Kyotaro Horiguchi wrote:
I thought that we have decided not to do that, but I coundn't find any
discussion about it in the ML archive. Anyway, I think it is great
that we have that option.No objections from here to make that optional. It's been argued for a
couple of times that leaving the archive history files is good for
debugging, but I can also get why one would one to clean up all the
files older than a WAL retention policy. Even if these are just few
bytes, it can be noisy for the eye to see a large accumulation of
history files mixed with the WAL segments.
--
Michael
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v2-0001-Allow-pg_archivecleanup-to-remove-backuphistoryfile.patchtext/x-diff; name=v2-0001-Allow-pg_archivecleanup-to-remove-backuphistoryfile.patchDownload
From 7d44134e5de930ce04819285a5b7359e370708d4 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 9 May 2023 21:52:01 +0900
Subject: [PATCH v2] Allow pg_archivecleanup to remove backup history files
Add new option -b to remove files including backup history files older
than oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 11 ++++-
.../t/010_pg_archivecleanup.pl | 45 +++++++++++++++++--
3 files changed, 62 insertions(+), 5 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..6cb565156f 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -93,6 +93,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<variablelist>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <listitem>
+ <para>
+ Remove files including backup history files, whose suffix is <filename>.backup</filename>.
+ Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file,
+ specified file is kept and only preceding WAL files and backup history files are removed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-d</option></term>
<listitem>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..ae6ae76f08 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool removeBackupHistoryFile = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -118,7 +120,8 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
+ if (((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) ||
+ (removeBackupHistoryFile && IsBackupHistoryFileName(walfile))) &&
strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
{
char WALFilePath[MAXPGPATH * 2]; /* the file path
@@ -252,6 +255,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b remove files including backup history files\n"));
printf(_(" -d generate debug output (verbose mode)\n"));
printf(_(" -n dry run, show the names of the files that would be removed\n"));
printf(_(" -V, --version output version information, then exit\n"));
@@ -294,10 +298,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt(argc, argv, "bdnx:")) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ removeBackupHistoryFile = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index 76321d1284..afd2ff6209 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -16,9 +16,14 @@ my @walfiles = (
'00000001000000370000000C.gz', '00000001000000370000000D',
'00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_backuphistoryfile = (
+ '000000020000003800000007.00000028.backup', '000000020000003800000008',
+ '000000020000003800000009',
+ '00000002000000380000000A', '00000002000000380000000B.007C9330.backup');
+
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_, 'unrelated_file')
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +32,7 @@ sub create_files
return;
}
-create_files();
+create_files(@walfiles);
command_fails_like(
['pg_archivecleanup'],
@@ -76,7 +81,7 @@ sub run_check
my ($suffix, $test_name) = @_;
- create_files();
+ create_files(@walfiles);
command_ok(
[
@@ -98,8 +103,42 @@ sub run_check
return;
}
+sub remove_backuphistoryfile_run_check
+{
+ local $Test::Builder::Level = $Test::Builder::Level + 1;
+
+ my ($suffix, $test_name) = @_;
+
+ create_files(@walfiles_with_backuphistoryfile);
+
+ command_ok(
+ [
+ 'pg_archivecleanup', '-b', $tempdir,
+ $walfiles_with_backuphistoryfile[2] . $suffix
+ ],
+ "$test_name: remove_backuphistoryfile_runs");
+
+ ok(!-f "$tempdir/$walfiles_with_backuphistoryfile[0]",
+ "$test_name: first .backup file was cleaned up");
+ ok(!-f "$tempdir/$walfiles_with_backuphistoryfile[1]",
+ "$test_name: second older WAL file was cleaned up");
+ ok(-f "$tempdir/$walfiles_with_backuphistoryfile[2]",
+ "$test_name: restartfile was not cleaned up");
+ ok(-f "$tempdir/$walfiles_with_backuphistoryfile[3]",
+ "$test_name: newer WAL file was not cleaned up");
+ ok(-f "$tempdir/$walfiles_with_backuphistoryfile[4]",
+ "$test_name: newer .backup file was not cleaned up");
+ ok(-f "$tempdir/unrelated_file",
+ "$test_name: unrelated file was not cleaned up");
+ return;
+}
+
run_check('', 'pg_archivecleanup');
run_check('.partial', 'pg_archivecleanup with .partial file');
run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+remove_backuphistoryfile_run_check('', 'pg_archivecleanup -b');
+remove_backuphistoryfile_run_check('.00000020.backup',
+ 'pg_archivecleanup -b with .backup file');
+
done_testing();
base-commit: c5b7f67fcc8c4a01c82660eb0996a3c697fac283
--
2.25.1
On Tue, May 9, 2023 at 7:03 PM torikoshia <torikoshia@oss.nttdata.com> wrote:
Attached a patch with documentation and regression tests.
Thanks. I think pg_archivecleanup cleaning up history files makes it a
complete feature as there's no need to write custom code/scripts over
and above what pg_archivecleanup provides. It will help those who are
using pg_archivecleanup for cleaning up older WAL files, say from
their archive location.
Just curious to know the driving point behind this proposal - is
pg_archivecleanup deployed in production that was unable to clean up
the history files and there were many such history files left? It will
help us know how pg_archivecleanup is being used.
I'm wondering if making -x generic with '-x' '.backup', is simpler
than adding another option?
Comments on the patch:
1. Why just only the backup history files? Why not remove the timeline
history files too? Is it because there may not be as many tli switches
happening as backups?
2.+sub remove_backuphistoryfile_run_check
+{
Why to invent a new function when run_check() can be made generic with
few arguments passed? For instance, run_check() can receive
pg_archivecleanup command args, what to use for create_files(), in the
error condition if the pg_archivecleanup command args contain 'b',
then use a different message "$test_name: first older WAL file was
cleaned up" or "$test_name: first .backup file was cleaned up".
Otherwise, just modify the messages to be:
"$test_name: first older file %s was cleaned up", $files[0]);
"$test_name: second older file %s was cleaned up", $files[1]);
"$test_name: restart file %s was not cleaned up", $files[2]);
"$test_name: newer file %s was not cleaned up", $files[3]);
"$test_name: unrelated file %s was not cleaned up", $files[4]);
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On 2023-05-10 17:52, Bharath Rupireddy wrote:
Thanks for your comments!
Just curious to know the driving point behind this proposal - is
pg_archivecleanup deployed in production that was unable to clean up
the history files and there were many such history files left? It will
help us know how pg_archivecleanup is being used.
Yes.
Just curious to know the driving point behind this proposal - is
pg_archivecleanup deployed in production that was unable to clean up
the history files and there were many such history files left? It will
help us know how pg_archivecleanup is being used.I'm wondering if making -x generic with '-x' '.backup', is simpler
than adding another option?
Since according to the current semantics, deleting backup history files
with -x demands not '-x .backup' but '-x .007C9330.backup' when the file
name is 0000000100001234000055CD.007C9330.backup, it needs special
treatment for backup history files, right?
I think it would be intuitive and easier to remember than new option.
I was a little concerned about what to do when deleting both the files
ending in .gz and backup history files.
Is making it possible to specify both "-x .backup" and "-x .gz" the way
to go?
I also concerned someone might add ".backup" to WAL files, but does that
usually not happen?
Comments on the patch:
1. Why just only the backup history files? Why not remove the timeline
history files too? Is it because there may not be as many tli switches
happening as backups?
Yeah, do you think we should also add logic for '-x .history'?
2.+sub remove_backuphistoryfile_run_check
+{
Why to invent a new function when run_check() can be made generic with
few arguments passed?
Thanks, I'm going to reconsider it.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
On Fri, May 12, 2023 at 05:53:45PM +0900, torikoshia wrote:
On 2023-05-10 17:52, Bharath Rupireddy wrote:
I was a little concerned about what to do when deleting both the files
ending in .gz and backup history files.
Is making it possible to specify both "-x .backup" and "-x .gz" the way to
go?I also concerned someone might add ".backup" to WAL files, but does that
usually not happen?
Depends on the archive command, of course. I have seen code using
this suffix for some segment names in the past, FWIW.
Comments on the patch:
1. Why just only the backup history files? Why not remove the timeline
history files too? Is it because there may not be as many tli switches
happening as backups?Yeah, do you think we should also add logic for '-x .history'?
Timeline history files can be critical pieces when it comes to
assigning a TLI as these could be retrieved by a restore_command
during recovery for a TLI jump or just assign a new TLI number at the
end of recovery, still you could presumably remove the TLI history
files that are older than the TLI the segment defined refers too.
(pg_archivecleanup has no specific logic to look at the history with
child TLIs for example, to keep it simple, and I'd rather keep it this
way). There may be an argument for making that optional, of course,
but it does not strike me as really necessary compared to the backup
history files which are just around for debugging purposes.
2.+sub remove_backuphistoryfile_run_check
+{
Why to invent a new function when run_check() can be made generic with
few arguments passed?Thanks, I'm going to reconsider it.
+ <para>
+ Remove files including backup history files, whose suffix is <filename>.backup</filename>.
+ Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file,
+ specified file is kept and only preceding WAL files and backup history files are removed.
+ </para>
This addition to the documentation looks unprecise to me. Backup
history files have a more complex format than just the .backup
suffix, and this is documented in backup.sgml.
How about plugging in some long options, and use something more
explicit like --clean-backup-history?
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
+ if (((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) ||
+ (removeBackupHistoryFile && IsBackupHistoryFileName(walfile))) &&
strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
Could it be a bit cleaner to split this check in two, saving one level
of indentation on the way for its most inner loop? I would imagine
something like:
/* Check file name */
if (!IsXLogFileName(walfile) &&
!IsPartialXLogFileName(walfile))
continue;
/* Check cutoff point */
if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
continue;
//rest of the code doing the unlinks.
--
Michael
On 2023-05-15 09:18, Michael Paquier wrote:
On Fri, May 12, 2023 at 05:53:45PM +0900, torikoshia wrote:
On 2023-05-10 17:52, Bharath Rupireddy wrote:
I was a little concerned about what to do when deleting both the files
ending in .gz and backup history files.
Is making it possible to specify both "-x .backup" and "-x .gz" the
way to
go?I also concerned someone might add ".backup" to WAL files, but does
that
usually not happen?Depends on the archive command, of course. I have seen code using
this suffix for some segment names in the past, FWIW.
Thanks for the information.
I'm going to stop adding special logic for "-x .backup" and add a new
option for removing backup history files.
Comments on the patch:
1. Why just only the backup history files? Why not remove the
timeline
history files too? Is it because there may not be as many tli
switches
happening as backups?Yeah, do you think we should also add logic for '-x .history'?
Timeline history files can be critical pieces when it comes to
assigning a TLI as these could be retrieved by a restore_command
during recovery for a TLI jump or just assign a new TLI number at the
end of recovery, still you could presumably remove the TLI history
files that are older than the TLI the segment defined refers too.
(pg_archivecleanup has no specific logic to look at the history with
child TLIs for example, to keep it simple, and I'd rather keep it this
way). There may be an argument for making that optional, of course,
but it does not strike me as really necessary compared to the backup
history files which are just around for debugging purposes.
Agreed.
2.+sub remove_backuphistoryfile_run_check
+{
Why to invent a new function when run_check() can be made generic
with
few arguments passed?Thanks, I'm going to reconsider it.
+ <para> + Remove files including backup history files, whose suffix is <filename>.backup</filename>. + Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file, + specified file is kept and only preceding WAL files and backup history files are removed. + </para>This addition to the documentation looks unprecise to me. Backup
history files have a more complex format than just the .backup
suffix, and this is documented in backup.sgml.
I'm going to remove the explanation for the backup history files and
just add the hyperlink to the original explanation in backup.sgml.
How about plugging in some long options, and use something more
explicit like --clean-backup-history?
Agreed.
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) && + if (((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) || + (removeBackupHistoryFile && IsBackupHistoryFileName(walfile))) && strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)Could it be a bit cleaner to split this check in two, saving one level
of indentation on the way for its most inner loop? I would imagine
something like:
/* Check file name */
if (!IsXLogFileName(walfile) &&
!IsPartialXLogFileName(walfile))
continue;
/* Check cutoff point */
if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
continue;
//rest of the code doing the unlinks.
--
Thanks, that looks better.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
On Mon, May 15, 2023 at 03:16:46PM +0900, torikoshia wrote:
On 2023-05-15 09:18, Michael Paquier wrote:
How about plugging in some long options, and use something more
explicit like --clean-backup-history?Agreed.
If you begin to implement that, it seems to me that this should be
shaped with a first separate patch that refactors the code to use
getopt_long(), and a second patch for the proposed feature that builds
on top of it.
--
Michael
On 2023-05-15 15:22, Michael Paquier wrote:
On Mon, May 15, 2023 at 03:16:46PM +0900, torikoshia wrote:
On 2023-05-15 09:18, Michael Paquier wrote:
How about plugging in some long options, and use something more
explicit like --clean-backup-history?Agreed.
If you begin to implement that, it seems to me that this should be
shaped with a first separate patch that refactors the code to use
getopt_long(), and a second patch for the proposed feature that builds
on top of it.
Thanks for your advice, attached patches.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v3-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v3-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 814b7351f14626f02c13b21d1a6737461117e5d0 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 22 May 2023 17:37:25 +0900
Subject: [PATCH v3 1/2] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 20 ++++++++++++-------
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..62914bdfa7 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,11 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +274,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +300,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
base-commit: ac298d3cb56b015acd40d2e015e07a87d8aff124
--
2.39.2
v3-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v3-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From 6017a774691cf52f7f51b817dee26db3bc7879c0 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 22 May 2023 17:41:57 +0900
Subject: [PATCH v3 2/2] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 13 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 88 +++++++++++--------
.../t/010_pg_archivecleanup.pl | 36 +++++---
3 files changed, 91 insertions(+), 46 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..e0efabd989 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,19 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove files including backup history file.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file,
+ specified file is kept and only preceding WAL files and backup history files are removed.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 62914bdfa7..3fd4a441b2 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
++ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,6 +99,8 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
* Truncation is essentially harmless, because we skip names of
* length other than XLOG_FNAME_LEN. (In principle, one could use
@@ -106,6 +110,19 @@ CleanupPriorWALFiles(void)
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ (!cleanBackupHistory || !IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +135,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -252,11 +265,12 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d, --debug generate debug output (verbose mode)\n"));
- printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -275,6 +289,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +315,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..544762ae4f 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -14,6 +14,7 @@ my $tempdir = PostgreSQL::Test::Utils::tempdir;
my @walfiles = (
'00000001000000370000000C.gz', '00000001000000370000000D',
+ '00000001000000370000000D.00000028.backup',
'00000001000000370000000E', '00000001000000370000000F.partial',);
sub create_files
@@ -59,7 +60,7 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles[3] ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
@@ -76,32 +77,45 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($suffix, $test_name, @options) = @_;
create_files();
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $walfiles[3] . $suffix
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
+ if (grep {$_ eq '-x.gz'} @options) {
+ ok(!-f "$tempdir/$walfiles[0]",
+ "$test_name: first older WAL file with .gz was cleaned up");
+ } else {
+ ok(-f "$tempdir/$walfiles[0]",
+ "$test_name: first older WAL file with .gz was not cleaned up");
+ }
ok(!-f "$tempdir/$walfiles[1]",
"$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
+ if (grep {$_ eq '-b'} @options) {
+ ok(!-f "$tempdir/$walfiles[2]",
+ "$test_name: Backup history file was cleaned up");
+ } else {
+ ok(-f "$tempdir/$walfiles[2]",
+ "$test_name: Backup history file was not cleaned up");
+ }
ok(-f "$tempdir/$walfiles[3]",
+ "$test_name: restartfile was not cleaned up");
+ ok(-f "$tempdir/$walfiles[4]",
"$test_name: newer WAL file was not cleaned up");
ok(-f "$tempdir/unrelated_file",
"$test_name: unrelated file was not cleaned up");
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check('', 'pg_archivecleanup', '-x.gz');
+run_check('.partial', 'pg_archivecleanup with .partial file', '-x.gz');
+run_check('.00000020.backup', 'pg_archivecleanup with .backup file', '-x.gz');
+run_check('', 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
On Mon, May 22, 2023 at 06:24:49PM +0900, torikoshia wrote:
Thanks for your advice, attached patches.
0001 looks OK, thanks!
+ Remove files including backup history file.
This could be reworded as "Remove backup history files.", I assume.
+ Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file,
+ specified file is kept and only preceding WAL files and backup history files are removed.
The same thing is described at the bottom of the documentation, so it
does not seem necessary here.
- printf(_(" -d, --debug generate debug output (verbose mode)\n"));
- printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
Perhaps this indentation had better be adjusted in 0001, no big deal
either way.
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
+ if (grep {$_ eq '-x.gz'} @options) {
+ ok(!-f "$tempdir/$walfiles[0]",
+ "$test_name: first older WAL file with .gz was cleaned up");
+ } else {
+ ok(-f "$tempdir/$walfiles[0]",
+ "$test_name: first older WAL file with .gz was not cleaned up");
+ }
[...]
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
+ if (grep {$_ eq '-b'} @options) {
+ ok(!-f "$tempdir/$walfiles[2]",
+ "$test_name: Backup history file was cleaned up");
+ } else {
+ ok(-f "$tempdir/$walfiles[2]",
+ "$test_name: Backup history file was not cleaned up");
+ }
That's over-complicated, caused by the fact that all the tests rely on
the same list of WAL files to create, aka @walfiles defined at the
beginning of the script. Wouldn't it be cleaner to handle the fake
file creations with more than one global list, before each command
run? The list of files to be created could just be passed down as an
argument of run_check(), for example, before cleaning up the contents
for the next command.
--
Michael
On 2023-05-24 10:26, Michael Paquier wrote:
On Mon, May 22, 2023 at 06:24:49PM +0900, torikoshia wrote:
Thanks for your advice, attached patches.
0001 looks OK, thanks!
+ Remove files including backup history file.
This could be reworded as "Remove backup history files.", I assume.
+ Note that when <replaceable>oldestkeptwalfile</replaceable> is a backup history file, + specified file is kept and only preceding WAL files and backup history files are removed.The same thing is described at the bottom of the documentation, so it
does not seem necessary here.- printf(_(" -d, --debug generate debug output (verbose mode)\n")); - printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n")); - printf(_(" -V, --version output version information, then exit\n")); - printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n")); - printf(_(" -?, --help show this help, then exit\n")); + printf(_(" -d, --debug generate debug output (verbose mode)\n")); + printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n")); + printf(_(" -b, --clean-backup-history clean up files including backup history files\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n")); + printf(_(" -?, --help show this help, then exit\n"));Perhaps this indentation had better be adjusted in 0001, no big deal
either way.- ok(!-f "$tempdir/$walfiles[0]", - "$test_name: first older WAL file was cleaned up"); + if (grep {$_ eq '-x.gz'} @options) { + ok(!-f "$tempdir/$walfiles[0]", + "$test_name: first older WAL file with .gz was cleaned up"); + } else { + ok(-f "$tempdir/$walfiles[0]", + "$test_name: first older WAL file with .gz was not cleaned up"); + } [...] - ok(-f "$tempdir/$walfiles[2]", - "$test_name: restartfile was not cleaned up"); + if (grep {$_ eq '-b'} @options) { + ok(!-f "$tempdir/$walfiles[2]", + "$test_name: Backup history file was cleaned up"); + } else { + ok(-f "$tempdir/$walfiles[2]", + "$test_name: Backup history file was not cleaned up"); + }That's over-complicated, caused by the fact that all the tests rely on
the same list of WAL files to create, aka @walfiles defined at the
beginning of the script. Wouldn't it be cleaner to handle the fake
file creations with more than one global list, before each command
run? The list of files to be created could just be passed down as an
argument of run_check(), for example, before cleaning up the contents
for the next command.
--
Michael
Thanks for reviewing!
Updated patches according to your comment.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v4-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v4-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 024b54d5d436bb88bb80bf6b316f697eb96b53f1 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 25 May 2023 23:13:12 +0900
Subject: [PATCH v4 1/2] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 20 ++++++++++++-------
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..707d7d54cf 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,11 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +274,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +300,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v4-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v4-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From 23d20f30eb0fa6daed1f5e631c5b3bedaeae623d Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 25 May 2023 23:25:28 +0900
Subject: [PATCH v4 2/2] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 78 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 46 ++++++-----
3 files changed, 87 insertions(+), 48 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 707d7d54cf..3fd4a441b2 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,6 +99,8 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
* Truncation is essentially harmless, because we skip names of
* length other than XLOG_FNAME_LEN. (In principle, one could use
@@ -106,6 +110,19 @@ CleanupPriorWALFiles(void)
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ (!cleanBackupHistory || !IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +135,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -254,6 +267,7 @@ usage(void)
printf(_("\nOptions:\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -275,6 +289,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +315,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..ee2f714c2a 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,18 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
+my @walfiles_with_gz = (
'00000001000000370000000C.gz', '00000001000000370000000D',
'00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_backup = (
+ '00000001000000370000000D',
+ '00000001000000370000000D.00000028.backup',
+ '00000001000000370000000E', '00000001000000370000000F.partial',);
+
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_, 'unrelated_file')
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +32,7 @@ sub create_files
return;
}
-create_files();
+create_files(@walfiles_with_gz);
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +64,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2] ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1].*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (@walfiles_with_gz)
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +81,37 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($walfiles, $suffix, $test_name, @options) = @_;
- create_files();
+ create_files(\@$walfiles);
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ @$walfiles[2] . $suffix
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
+ ok(!-f "$tempdir/@$walfiles[0]",
"$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
+ ok(!-f "$tempdir/@$walfiles[1]",
+ "$test_name: second older WAL/backup history file was cleaned up");
+ ok(-f "$tempdir/@$walfiles[2]",
"$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
+ ok(-f "$tempdir/@$walfiles[3]",
"$test_name: newer WAL file was not cleaned up");
ok(-f "$tempdir/unrelated_file",
"$test_name: unrelated file was not cleaned up");
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
At Thu, 25 May 2023 23:51:18 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
Updated patches according to your comment.
+ printf(_(" -x --strip-extension=EXT clean up files if they have this extension\n"));
Perhaps a comma is needed after "-x". The apparent inconsistency
between the long name and the description is perplexing. I think it
might be more suitable to reword the descrition to "ignore this
extension while identifying files for cleanup" or something along
those lines..., and then name the option as "--ignore-extension".
The patch leaves the code.
* Truncation is essentially harmless, because we skip names of
* length other than XLOG_FNAME_LEN. (In principle, one could use
* a 1000-character additional_ext and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
The comment is no longer correct about the file name length.
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ (!cleanBackupHistory || !IsBackupHistoryFileName(walfile)))
+ continue;
I'm not certain about the others, but I see this a tad tricky to
understand at first glance. Here's how I would phrase it. (A nearby
comment might require a tweak if we go ahead with this change.)
if (!(IsXLogFileName(walfile) || IsPartialXLogFileName(walfile) ||
(cleanBackupHistory && IsBackupHistoryFileName(walfile))))
or
if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
!(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
By the way, this patch reduces the nesting level. If we go that
direction, the following structure could be reworked similarly, and I
believe it results in a more standard structure.
if ((xldir = opendir(archiveLocation)) != NULL)
{
...
}
else
pg_fatal("could not open archive location..
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Fri, May 26, 2023 at 10:07:48AM +0900, Kyotaro Horiguchi wrote:
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) && + (!cleanBackupHistory || !IsBackupHistoryFileName(walfile))) + continue;I'm not certain about the others, but I see this a tad tricky to
understand at first glance. Here's how I would phrase it. (A nearby
comment might require a tweak if we go ahead with this change.)if (!(IsXLogFileName(walfile) || IsPartialXLogFileName(walfile) ||
(cleanBackupHistory && IsBackupHistoryFileName(walfile))))
or
if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
!(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
FWIW, I am OK with what's written in the patch, but it is true that
your second suggestion makes things a bit easier to read, perhaps.
--
Michael
On Thu, May 25, 2023 at 11:51:18PM +0900, torikoshia wrote:
Updated patches according to your comment.
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
+ ok(!-f "$tempdir/@$walfiles[1]",
+ "$test_name: second older WAL/backup history file was cleaned up");
+ ok(-f "$tempdir/@$walfiles[2]",
This is still a bit confusing, because as designed the test has a
dependency on the number of elements present in the list, and the
description of the test may not refer to what's actually used (the
second element of each list is either a WAL segment or a backup
history file). I think that I would just rewrite that so as we have a
list of WAL segments in an array with the expected result associated
to each one of them. Basically, something like that:
my @wal_segments = (
{ name => "SEGMENT1", present => 0 },
{ name => "BACKUPFILE1", present => 1 },
{ name => "SEGMENT2", present => 0 });
Then the last part of run_check() would loop through all the elements
listed.
--
Michael
On 2023-05-26 10:07, Kyotaro Horiguchi wrote:
Thanks for your review!
+ printf(_(" -x --strip-extension=EXT clean up files if they have
this extension\n"));Perhaps a comma is needed after "-x".
That's an oversight. Modified it.
The apparent inconsistency
between the long name and the description is perplexing. I think it
might be more suitable to reword the descrition to "ignore this
extension while identifying files for cleanup" or something along
those lines..., and then name the option as "--ignore-extension".
I used 'strip' since it is used in the manual as below:
| -x extension
| Provide an extension that will be stripped from all file names before
deciding if they should be deleted
I think using the same verb both in long name option and in the manual
is natural.
How about something like this?
| printf(_(" -x, --strip-extension=EXT strip this extention before
identifying files fo clean up\n"));
The patch leaves the code.
* Truncation is essentially harmless, because we skip names of
* length other than XLOG_FNAME_LEN. (In principle, one could use
* a 1000-character additional_ext and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);The comment is no longer correct about the file name length.
Yeah.
considering parital WAL, it would be not correct even before applying
the patch.
I modifiied the comments as below:
| * Truncation is essentially harmless, because we check the file
| * format including the length immediately after this.
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) && + (!cleanBackupHistory || !IsBackupHistoryFileName(walfile))) + continue;I'm not certain about the others, but I see this a tad tricky to
understand at first glance. Here's how I would phrase it. (A nearby
comment might require a tweak if we go ahead with this change.)if (!(IsXLogFileName(walfile) || IsPartialXLogFileName(walfile) ||
(cleanBackupHistory && IsBackupHistoryFileName(walfile))))or
if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
!(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
Thanks for the suggestion, I used the second one.
By the way, this patch reduces the nesting level. If we go that
direction, the following structure could be reworked similarly, and I
believe it results in a more standard structure.if ((xldir = opendir(archiveLocation)) != NULL)
{
...
}
else
pg_fatal("could not open archive location..
Agreed. Attached 0003 patch for this.
On 2023-05-26 14:19, Michael Paquier wrote:
On Thu, May 25, 2023 at 11:51:18PM +0900, torikoshia wrote:
Updated patches according to your comment.
- ok(!-f "$tempdir/$walfiles[1]", - "$test_name: second older WAL file was cleaned up"); - ok(-f "$tempdir/$walfiles[2]", + ok(!-f "$tempdir/@$walfiles[1]", + "$test_name: second older WAL/backup history file was cleaned up"); + ok(-f "$tempdir/@$walfiles[2]",This is still a bit confusing, because as designed the test has a
dependency on the number of elements present in the list, and the
description of the test may not refer to what's actually used (the
second element of each list is either a WAL segment or a backup
history file). I think that I would just rewrite that so as we have a
list of WAL segments in an array with the expected result associated
to each one of them. Basically, something like that:
my @wal_segments = (
{ name => "SEGMENT1", present => 0 },
{ name => "BACKUPFILE1", present => 1 },
{ name => "SEGMENT2", present => 0 });Then the last part of run_check() would loop through all the elements
listed.
Thanks.
Update the patch according to the advice.
I also changed the parameter of run_check() from specifying extension of
oldestkeptwalfile to oldestkeptwalfile itself.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v5-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v5-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 06b5664d1093aba445332ad5420bd442b99a8f98 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 30 May 2023 20:44:37 +0900
Subject: [PATCH v5 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 20 ++++++++++++-------
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..be62ffe75d 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,11 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +274,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +300,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v5-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v5-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From ef234fe06f32b0b3b0b3d54cefc78ee7fbf40328 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 30 May 2023 20:46:42 +0900
Subject: [PATCH v5 2/3] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 85 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 79 +++++++++++------
3 files changed, 117 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index be62ffe75d..a7f77d9158 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,15 +99,31 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +136,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -254,6 +268,7 @@ usage(void)
printf(_("\nOptions:\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -275,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
v5-0003-Refactored-to-reduce-the-nesting-level.patchtext/x-diff; name=v5-0003-Refactored-to-reduce-the-nesting-level.patchDownload
From bef376b2766923339dedbf3fae9e516262b433a1 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 31 May 2023 10:09:59 +0900
Subject: [PATCH v5 3/3] Refactored to reduce the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 146 +++++++++---------
1 file changed, 72 insertions(+), 74 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index a7f77d9158..f475177edb 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -95,87 +95,85 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
+ if ((xldir = opendir(archiveLocation)) == NULL)
+ pg_fatal("could not open archive location \"%s\": %m",
+ archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
{
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
- /*
- * Truncation is essentially harmless, because we check the file
- * format including the length immediately after this.
- * (In principle, one could use a 1000-character additional_ext
- * and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
/*
- * Check file name.
- *
- * We skip files which are not WAL file or partial WAL file.
- * Also we skip backup history files when --clean-backup-history
- * is not specified.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
- !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
- continue;
-
- /*
- * Check cutoff point.
- *
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
- continue;
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
- else
- pg_fatal("could not open archive location \"%s\": %m",
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
archiveLocation);
}
--
2.39.2
On 2023/05/31 10:51, torikoshia wrote:
Update the patch according to the advice.
Thanks for updating the patches! I have small comments regarding 0002 patch.
+ <para>
+ Remove backup history files.
Isn't it better to document clearly which backup history files to be removed? For example, "In addition to removing WAL files, remove backup history files with prefixes logically preceding the oldestkeptwalfile.".
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
Shouldn't -b option be placed in alphabetical order?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On Fri, Jun 09, 2023 at 12:32:15AM +0900, Fujii Masao wrote:
+ <para>
+ Remove backup history files.Isn't it better to document clearly which backup history files to be removed? For example, "In addition to removing WAL files, remove backup history files with prefixes logically preceding the oldestkeptwalfile.".
I've written about this part at the beginning of this one, where this
sounds like a duplicated description of the Description section:
/messages/by-id/ZG1nq13v411y4TFL@paquier.xyz
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));Shouldn't -b option be placed in alphabetical order?
+1.
--
Michael
On 2023-06-12 16:33, Michael Paquier wrote:
On Fri, Jun 09, 2023 at 12:32:15AM +0900, Fujii Masao wrote:
Thanks for reviewing!
printf(_(" -n, --dry-run dry run, show the names of
the files that would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including
backup history files\n"));Shouldn't -b option be placed in alphabetical order?
+1.
Modified the place.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v6-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v6-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 06b5664d1093aba445332ad5420bd442b99a8f98 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 14 Jun 2023 00:30:37 +0900
Subject: [PATCH v5 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 20 ++++++++++++-------
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..be62ffe75d 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,11 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +274,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +300,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v6-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v6-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From ef234fe06f32b0b3b0b3d54cefc78ee7fbf40328 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 14 Jun 2023 00:31:39 +0900
Subject: [PATCH v5 2/3] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 85 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 79 +++++++++++------
3 files changed, 117 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index be62ffe75d..a7f77d9158 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,15 +99,31 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +136,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -254,6 +268,7 @@ usage(void)
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -275,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
v6-0003-Refactored-to-reduce-the-nesting-level.patchtext/x-diff; name=v6-0003-Refactored-to-reduce-the-nesting-level.patchDownload
From bef376b2766923339dedbf3fae9e516262b433a1 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 14 Jun 2023 00:33:10 +0900
Subject: [PATCH v5 3/3] Refactored to reduce the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 146 +++++++++---------
1 file changed, 72 insertions(+), 74 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index a7f77d9158..f475177edb 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -95,87 +95,85 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
+ if ((xldir = opendir(archiveLocation)) == NULL)
+ pg_fatal("could not open archive location \"%s\": %m",
+ archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
{
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
- /*
- * Truncation is essentially harmless, because we check the file
- * format including the length immediately after this.
- * (In principle, one could use a 1000-character additional_ext
- * and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
/*
- * Check file name.
- *
- * We skip files which are not WAL file or partial WAL file.
- * Also we skip backup history files when --clean-backup-history
- * is not specified.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
- !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
- continue;
-
- /*
- * Check cutoff point.
- *
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
- continue;
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
- else
- pg_fatal("could not open archive location \"%s\": %m",
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
archiveLocation);
}
--
2.39.2
At Wed, 14 Jun 2023 00:49:39 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
On 2023-06-12 16:33, Michael Paquier wrote:
On Fri, Jun 09, 2023 at 12:32:15AM +0900, Fujii Masao wrote:
Thanks for reviewing!
printf(_(" -n, --dry-run dry run, show the names of the files that
would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including
backup history files\n"));
Shouldn't -b option be placed in alphabetical order?+1.
Modified the place.
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
After this change, some of these lines corss the boundary of the 80
columns width. (is that policy viable noadays? I am usually working
using terminal windows with such a width..) It's somewhat unrelated to
this patch, but a help line a few lines further down also exceeds the
width. We could shorten it by removing the "/mnt/server" portion, but
I'm not sure if it's worth doing.
Or for use as a standalone archive cleaner:
e.g.
pg_archivecleanup /mnt/server/archiverdir 000000010000000000000010.00000020.backup
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n"));
s/fo/for/ ?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2023-06-15 15:20, Kyotaro Horiguchi wrote:
Thanks for your review!
At Wed, 14 Jun 2023 00:49:39 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inOn 2023-06-12 16:33, Michael Paquier wrote:
On Fri, Jun 09, 2023 at 12:32:15AM +0900, Fujii Masao wrote:
Thanks for reviewing!
printf(_(" -n, --dry-run dry run, show the names of the files that
would be removed\n"));
+ printf(_(" -b, --clean-backup-history clean up files including
backup history files\n"));
Shouldn't -b option be placed in alphabetical order?+1.
Modified the place.
- printf(_(" -d generate debug output (verbose mode)\n")); - printf(_(" -n dry run, show the names of the files that would be removed\n")); - printf(_(" -V, --version output version information, then exit\n")); - printf(_(" -x EXT clean up files if they have this extension\n")); - printf(_(" -?, --help show this help, then exit\n")); + printf(_(" -d, --debug generate debug output (verbose mode)\n")); + printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n")); + printf(_(" -V, --version output version information, then exit\n")); + printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n")); + printf(_(" -?, --help show this help, then exit\n"));After this change, some of these lines corss the boundary of the 80
columns width. (is that policy viable noadays? I am usually working
using terminal windows with such a width..) It's somewhat unrelated to
this patch, but a help line a few lines further down also exceeds the
width. We could shorten it by removing the "/mnt/server" portion, but
I'm not sure if it's worth doing.
I also highlight 80th column according to the wiki[1]https://wiki.postgresql.org/wiki/Configuring_vim_for_postgres_development -- Regards,.
Since usage() in other files like pg_rewind.c and initdb.c also
exceeded the 80th column, I thought that was something like a guide.
Or for use as a standalone archive cleaner:
e.g.
pg_archivecleanup /mnt/server/archiverdir
000000010000000000000010.00000020.backup+ printf(_(" -x, --strip-extension=EXT strip this extention before
identifying files fo clean up\n"));s/fo/for/ ?
Yeah, it's a typo. Fixed it.
[1]: https://wiki.postgresql.org/wiki/Configuring_vim_for_postgres_development -- Regards,
https://wiki.postgresql.org/wiki/Configuring_vim_for_postgres_development
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v7-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v7-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 06b5664d1093aba445332ad5420bd442b99a8f98 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 16 Jun 2023 21:13:02 +0900
Subject: [PATCH v7 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 20 ++++++++++++-------
2 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..be62ffe75d 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,11 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +274,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +300,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v7-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v7-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From ef234fe06f32b0b3b0b3d54cefc78ee7fbf40328 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 16 Jun 2023 21:15:23 +0900
Subject: [PATCH v7 2/3] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 85 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 79 +++++++++++------
3 files changed, 117 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index be62ffe75d..a7f77d9158 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,15 +99,31 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +136,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -254,6 +268,7 @@ usage(void)
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be removed\n"));
printf(_(" -V, --version output version information, then exit\n"));
printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for clean up\n"));
printf(_(" -?, --help show this help, then exit\n"));
@@ -275,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
v7-0003-Refactored-to-reduce-the-nesting-level.patchtext/x-diff; name=v7-0003-Refactored-to-reduce-the-nesting-level.patchDownload
From bef376b2766923339dedbf3fae9e516262b433a1 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Thu, 16 Jun 2023 21:23:02 +0900
Subject: [PATCH v7 3/3] Refactored to reduce the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 146 +++++++++---------
1 file changed, 72 insertions(+), 74 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index a7f77d9158..f475177edb 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -95,87 +95,85 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
+ if ((xldir = opendir(archiveLocation)) == NULL)
+ pg_fatal("could not open archive location \"%s\": %m",
+ archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
{
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
- /*
- * Truncation is essentially harmless, because we check the file
- * format including the length immediately after this.
- * (In principle, one could use a 1000-character additional_ext
- * and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
/*
- * Check file name.
- *
- * We skip files which are not WAL file or partial WAL file.
- * Also we skip backup history files when --clean-backup-history
- * is not specified.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
- !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
- continue;
-
- /*
- * Check cutoff point.
- *
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
- continue;
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
- else
- pg_fatal("could not open archive location \"%s\": %m",
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
archiveLocation);
}
--
2.39.2
At Thu, 15 Jun 2023 21:38:28 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
On 2023-06-15 15:20, Kyotaro Horiguchi wrote:
Thanks for your review!+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n")); + printf(_(" -?, --help show this help, then exit\n")); After this change, some of these lines corss the boundary of the 80 columns width. (is that policy viable noadays? I am usually working using terminal windows with such a width..) It's somewhat unrelated to this patch, but a help line a few lines further down also exceeds the width. We could shorten it by removing the "/mnt/server" portion, but I'm not sure if it's worth doing.I also highlight 80th column according to the wiki[1].
Since usage() in other files like pg_rewind.c and initdb.c also
exceeded the 80th column, I thought that was something like a guide.
I think the page is suggesting about program code, not the messages
that binaries print.
ASAICS the main section of the "pg_rewind --help" fits within 80
columns. However, "initdb --help" does output a few lines exceeding
the 80-column limit. Judging by the surrounding lines, I believe we're
still aiming to maintain these the given width. I think we need to fix
initdb in that regard.
[1]
https://wiki.postgresql.org/wiki/Configuring_vim_for_postgres_development
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
At Fri, 16 Jun 2023 11:22:31 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in
ASAICS the main section of the "pg_rewind --help" fits within 80
columns. However, "initdb --help" does output a few lines exceeding
the 80-column limit. Judging by the surrounding lines, I believe we're
still aiming to maintain these the given width. I think we need to fix
initdb in that regard.
Mmm, the message was introduced in 2012 by 8a02339e9b. I haven't
noticed this until now...
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2023-06-16 11:22, Kyotaro Horiguchi wrote:
At Thu, 15 Jun 2023 21:38:28 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inOn 2023-06-15 15:20, Kyotaro Horiguchi wrote:
Thanks for your review!+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files fo clean up\n")); + printf(_(" -?, --help show this help, then exit\n")); After this change, some of these lines corss the boundary of the 80 columns width. (is that policy viable noadays? I am usually working using terminal windows with such a width..) It's somewhat unrelated to this patch, but a help line a few lines further down also exceeds the width. We could shorten it by removing the "/mnt/server" portion, but I'm not sure if it's worth doing.I also highlight 80th column according to the wiki[1].
Since usage() in other files like pg_rewind.c and initdb.c also
exceeded the 80th column, I thought that was something like a guide.I think the page is suggesting about program code, not the messages
that binaries print.
Thanks, now I understand what you meant.
ASAICS the main section of the "pg_rewind --help" fits within 80
columns. However, "initdb --help" does output a few lines exceeding
the 80-column limit. Judging by the surrounding lines, I believe we're
still aiming to maintain these the given width. I think we need to fix
initdb in that regard.
Hmm, it seems some other commands also exceeds 80 columns:
pg_amcheck:
--skip=OPTION do NOT check "all-frozen" or
"all-visible" blocks
--startblock=BLOCK begin checking table(s) at the given
block number
--endblock=BLOCK check table(s) only up to the given
block number
--no-synchronized-snapshots do not use synchronized snapshots in
parallel jobs
pg_isready:
-t, --timeout=SECS seconds to wait when attempting connection, 0
disables (default: 3)
pg_receivewal:
--create-slot create a new replication slot (for the slot's
name see --slot)
--drop-slot drop the replication slot (for the slot's name
see --slot)
If you don't mind, I'm going to create another thread about this point.
I'll also discuss below line since it's unrelated to current thread
as you pointed out:
| pg_archivecleanup /mnt/server/archiverdir
000000010000000000000010.00000020.backup
Attached patch fixes the number of columns per row exceeding 80 by
changing to use getopt_long.
On 2023-06-16 11:30, Kyotaro Horiguchi wrote:
At Fri, 16 Jun 2023 11:22:31 +0900 (JST), Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote inASAICS the main section of the "pg_rewind --help" fits within 80
columns. However, "initdb --help" does output a few lines exceeding
the 80-column limit. Judging by the surrounding lines, I believe we're
still aiming to maintain these the given width. I think we need to fix
initdb in that regard.Mmm, the message was introduced in 2012 by 8a02339e9b. I haven't
noticed this until now...regards.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v8-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v8-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From 220bbd866158fd69a3a4affe73136f4699353ecd Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 09:46:41 +0900
Subject: [PATCH v8 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 22 +++++++++++++------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..fc0dca9856 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,13 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
+ " removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for\n"
+ " clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +276,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +302,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v8-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v8-0002-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From d065cf4981f827e1eea76577f92d39e3669dd816 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 09:51:12 +0900
Subject: [PATCH v8 2/3] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 85 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 79 +++++++++++------
3 files changed, 117 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index fc0dca9856..41880e42bb 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -97,15 +99,31 @@ CleanupPriorWALFiles(void)
{
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -118,39 +136,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -252,6 +266,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
" removed\n"));
@@ -277,6 +292,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -302,10 +318,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
v8-0003-Refactored-to-reduce-the-nesting-level.patchtext/x-diff; name=v8-0003-Refactored-to-reduce-the-nesting-level.patchDownload
From 806f1c5770763885137a4fccbfa3394f6cd5f8e9 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 09:58:08 +0900
Subject: [PATCH v8 3/3] Refactored to reduce the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 146 +++++++++---------
1 file changed, 72 insertions(+), 74 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 41880e42bb..ffb4a19e7f 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -95,87 +95,85 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
+ if ((xldir = opendir(archiveLocation)) == NULL)
+ pg_fatal("could not open archive location \"%s\": %m",
+ archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
{
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
- /*
- * Truncation is essentially harmless, because we check the file
- * format including the length immediately after this.
- * (In principle, one could use a 1000-character additional_ext
- * and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
/*
- * Check file name.
- *
- * We skip files which are not WAL file or partial WAL file.
- * Also we skip backup history files when --clean-backup-history
- * is not specified.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
- !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
- continue;
-
- /*
- * Check cutoff point.
- *
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
- continue;
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
- else
- pg_fatal("could not open archive location \"%s\": %m",
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
archiveLocation);
}
--
2.39.2
On Mon, Jun 19, 2023 at 11:24:29AM +0900, torikoshia wrote:
Thanks, now I understand what you meant.
If I may ask, why is the refactoring of 0003 done after the feature in
0002? Shouldn't the order be reversed? That would make for a cleaner
git history.
--
Michael
On 2023-06-19 14:37, Michael Paquier wrote:
On Mon, Jun 19, 2023 at 11:24:29AM +0900, torikoshia wrote:
Thanks, now I understand what you meant.
If I may ask, why is the refactoring of 0003 done after the feature in
0002? Shouldn't the order be reversed? That would make for a cleaner
git history.
--
Michael
Agreed.
Reversed the order of patches 0002 and 0003.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v9-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v9-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From a0770f668bf73477815fb44a7386d95ed91bdec5 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 20:42:07 +0900
Subject: [PATCH v9 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 22 +++++++++++++------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..fc0dca9856 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,13 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
+ " removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for\n"
+ " clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +276,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +302,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v9-0002-Refactored-to-reduce-the-nesting-level.patchtext/x-diff; name=v9-0002-Refactored-to-reduce-the-nesting-level.patchDownload
From e61734f9d2efd326bc259e3f869207404fe80143 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 20:47:53 +0900
Subject: [PATCH v9 2/3] Refactored to reduce the nesting level
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 114 +++++++++---------
1 file changed, 56 insertions(+), 58 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index fc0dca9856..950e27e63b 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -93,75 +93,73 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
+ if ((xldir = opendir(archiveLocation)) == NULL)
+ pg_fatal("could not open archive location \"%s\": %m",
+ archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ /*
+ * Truncation is essentially harmless, because we skip names of
+ * length other than XLOG_FNAME_LEN. (In principle, one could use
+ * a 1000-character additional_ext and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
+ strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
{
- /*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
- }
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
+ }
}
- else
- pg_fatal("could not open archive location \"%s\": %m",
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
archiveLocation);
}
--
2.39.2
v9-0003-Allow-pg_archivecleanup-to-remove-backup-history-.patchtext/x-diff; name=v9-0003-Allow-pg_archivecleanup-to-remove-backup-history-.patchDownload
From d938f968848b762d3665f7e7cbf6bf56574758d6 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 19 Jun 2023 20:54:04 +0900
Subject: [PATCH v9 3/3] Allow pg_archivecleanup to remove backup history files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 85 ++++++++++++-------
.../t/010_pg_archivecleanup.pl | 79 +++++++++++------
3 files changed, 117 insertions(+), 58 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 950e27e63b..ffb4a19e7f 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -99,15 +101,31 @@ CleanupPriorWALFiles(void)
while (errno = 0, (xlde = readdir(xldir)) != NULL)
{
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
/*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
* We ignore the timeline part of the XLOG segment identifiers in
* deciding whether a segment is still needed. This ensures that
* we won't prematurely remove a segment from a parent timeline.
@@ -120,39 +138,35 @@ CleanupPriorWALFiles(void)
* file. Note that this means files are not removed in the order
* they were originally written, in case this worries you.
*/
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
/*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
*/
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
}
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
}
if (errno)
@@ -250,6 +264,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
" removed\n"));
@@ -275,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -300,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
At Tue, 20 Jun 2023 22:27:36 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
On 2023-06-19 14:37, Michael Paquier wrote:
On Mon, Jun 19, 2023 at 11:24:29AM +0900, torikoshia wrote:
Thanks, now I understand what you meant.
If I may ask, why is the refactoring of 0003 done after the feature in
0002? Shouldn't the order be reversed? That would make for a cleaner
git history.
--
MichaelAgreed.
Reversed the order of patches 0002 and 0003.
Yeah, that is a possible division. However, I meant that we have room
to refactor and decrease the nesting level even further, considering
that 0003 already does this to some extent, when I suggested it. In
that sense, moving the nest-reduction part of 0003 into 0002 makes us
possible to focus on the point of this patch.
What do you think about the attached version?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
v10-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-patch; charset=us-asciiDownload
From 5777c28c01a6ca3bb48baf7dfabf6e720fcaaf5e Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 11:34:10 +0900
Subject: [PATCH v10 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 22 +++++++++++++------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..fc0dca9856 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,13 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
+ " removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for\n"
+ " clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +276,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +302,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.3
v10-0002-Preliminary-refactoring-for-a-subsequent-patch.patchtext/x-patch; charset=us-asciiDownload
From 060922adc1e529c7f6ce5a7bdf726d6f4acaeab4 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 11:34:53 +0900
Subject: [PATCH v10 2/3] Preliminary refactoring for a subsequent patch
This is a preparatory patch that doesn't introduce any functional
change. Instead, this carries out preliminary refactoring with the
goal of reducing the overall nesting level. This helps to prevent the
forthcoming patch from further deepening the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 144 +++++++++---------
1 file changed, 76 insertions(+), 68 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index fc0dca9856..f7269068cb 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -93,76 +93,84 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
- {
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
- {
- /*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
- /*
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
- }
- }
-
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
- }
- else
+ if ((xldir = opendir(archiveLocation)) == NULL)
pg_fatal("could not open archive location \"%s\": %m",
archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ {
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we skip names of
+ * length other than XLOG_FNAME_LEN. (In principle, one could use
+ * a 1000-character additional_ext and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
+ /*
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
+ */
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
+ }
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
+ }
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
+ archiveLocation);
}
/*
--
2.39.3
v10-0003-Allow-pg_archivecleanup-to-remove-backup-history.patchtext/x-patch; charset=us-asciiDownload
From 110aeb49d8fb0c50777e719ae75fd3faa4b759c3 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 11:36:28 +0900
Subject: [PATCH v10 3/3] Allow pg_archivecleanup to remove backup history
files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 19 +++--
.../t/010_pg_archivecleanup.pl | 79 +++++++++++++------
3 files changed, 79 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..8fac233db6 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -113,6 +113,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-V</option></term>
<term><option>--version</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index f7269068cb..ffb4a19e7f 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -102,9 +104,10 @@ CleanupPriorWALFiles(void)
char WALFilePath[MAXPGPATH * 2]; /* the file path
* including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
@@ -116,7 +119,8 @@ CleanupPriorWALFiles(void)
* Also we skip backup history files when --clean-backup-history
* is not specified.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
continue;
/*
@@ -260,6 +264,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
" removed\n"));
@@ -285,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -310,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.3
On 2023-06-21 11:59, Kyotaro Horiguchi wrote:
At Tue, 20 Jun 2023 22:27:36 +0900, torikoshia
<torikoshia@oss.nttdata.com> wrote inOn 2023-06-19 14:37, Michael Paquier wrote:
On Mon, Jun 19, 2023 at 11:24:29AM +0900, torikoshia wrote:
Thanks, now I understand what you meant.
If I may ask, why is the refactoring of 0003 done after the feature in
0002? Shouldn't the order be reversed? That would make for a cleaner
git history.
--
MichaelAgreed.
Reversed the order of patches 0002 and 0003.Yeah, that is a possible division. However, I meant that we have room
to refactor and decrease the nesting level even further, considering
that 0003 already does this to some extent, when I suggested it. In
that sense, moving the nest-reduction part of 0003 into 0002 makes us
possible to focus on the point of this patch.
Thanks for the comment, it seems better than v9 patch.
What do you think about the attached version?
--v10-0002-Preliminary-refactoring-for-a-subsequent-patch.patch
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ continue;
I think this comment should be located in 0003.
Attached updated patches.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v11-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v11-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From eb99f581b2990c80329f2baf96ed4d5e9c00dda5 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 21:42:08 +0900
Subject: [PATCH v11 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 22 +++++++++++++------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..fc0dca9856 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,13 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
+ " removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for\n"
+ " clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +276,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +302,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v11-0002-Preliminary-refactoring-for-a-subsequent-patch.patchtext/x-diff; name=v11-0002-Preliminary-refactoring-for-a-subsequent-patch.patchDownload
From d760567e6974a5b3618aba228f92c188529f6a75 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 22:20:21 +0900
Subject: [PATCH v11 2/3] Preliminary refactoring for a subsequent patch
This is a preparatory patch that doesn't introduce any functional
change. Instead, this carries out preliminary refactoring with the
goal of reducing the overall nesting level. This helps to prevent the
forthcoming patch from further deepening the nesting level.
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 142 +++++++++---------
1 file changed, 74 insertions(+), 68 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index fc0dca9856..982fe00f20 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -93,76 +93,82 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
- {
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
- {
- /*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
- /*
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
- }
- }
-
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
- }
- else
+ if ((xldir = opendir(archiveLocation)) == NULL)
pg_fatal("could not open archive location \"%s\": %m",
archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ {
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we skip names of
+ * length other than XLOG_FNAME_LEN. (In principle, one could use
+ * a 1000-character additional_ext and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ /*
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
+ */
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ continue;
+
+ /*
+ * Check cutoff point.
+ *
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
+ /*
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
+ */
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
+ }
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
+ }
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
+ archiveLocation);
}
/*
--
2.39.2
v11-0003-Allow-pg_archivecleanup-to-remove-backup-history.patchtext/x-diff; name=v11-0003-Allow-pg_archivecleanup-to-remove-backup-history.patchDownload
From 4228f489090161f2f968f83c32fd2e05e013136a Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 22:53:51 +0900
Subject: [PATCH v11 3/3] Allow pg_archivecleanup to remove backup history
files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 21 +++--
.../t/010_pg_archivecleanup.pl | 79 +++++++++++++------
3 files changed, 81 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..06c043f268 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -93,6 +93,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<variablelist>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files.
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-d</option></term>
<term><option>--debug</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 982fe00f20..1ed08518ca 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -102,9 +104,10 @@ CleanupPriorWALFiles(void)
char WALFilePath[MAXPGPATH * 2]; /* the file path
* including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
@@ -113,8 +116,11 @@ CleanupPriorWALFiles(void)
* Check file name.
*
* We skip files which are not WAL file or partial WAL file.
+ * Also we skip backup history files when --clean-backup-history
+ * is not specified.
*/
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
continue;
/*
@@ -258,6 +264,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
" removed\n"));
@@ -283,6 +290,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -308,10 +316,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files too */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..030a82e7fa 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,13 +12,23 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_with_backup = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (@_)
{
open my $file, '>', "$tempdir/$fn";
print $file 'CONTENT';
@@ -27,7 +37,18 @@ sub create_files
return;
}
-create_files();
+sub get_walfiles
+{
+ my @walfiles;
+
+ for my $walpair (@_)
+ {
+ push @walfiles, $walpair->{name};
+ }
+ return @walfiles;
+}
+
+create_files(get_walfiles(@walfiles_with_gz));
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +80,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[2]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (get_walfiles(@walfiles_with_gz))
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +97,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(get_walfiles(@$testdata));
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_with_backup, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
At Wed, 21 Jun 2023 23:41:33 +0900, torikoshia <torikoshia@oss.nttdata.com> wrote in
--v10-0002-Preliminary-refactoring-for-a-subsequent-patch.patch + * Also we skip backup history files when --clean-backup-history + * is not specified. + */ + if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile)) + continue;I think this comment should be located in 0003.
Auch! Right.
Attached updated patches.
Thanks!
0001:
Looks good to me.
0002:
+ * Check file name.
+ *
+ * We skip files which are not WAL file or partial WAL file.
There's no need to spend this many lines describing this, and it's
suggested to avoid restating what the code does. I think a comment
might not be necessary. But if we decide to have one, it could look
like this:
/* we're only removing specific types of files */
Other than that, it looks good to me.
0003:
+ <para>
+ Remove backup history files.
I might be overthinking it, but I'd phrase it as, "Remove backup
history files *as well*.". (The --help message describes the same
thing using "including".)
+ For details about backup history file, please refer to the <xref linkend="backup-base-backup"/>.
We usually write this as simple as "See <xref...> for details (of the
backup history files)" or "Refer to <xref..> for more information
(about the backup history files)." or such like... (I think)
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
The indentation appears to be inconsistent.
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we check the file
+ * format including the length immediately after this.
+ * (In principle, one could use a 1000-character additional_ext
+ * and get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
The revised comment seems to drift from the original point. Except for
a potential exception by a too-long addition_ext, the original comment
asserted that the name truncation was safe since it wouldn't affect
the files we're removing. In other words, it asserted that the
filenames to be removed won't be truncated and any actual truncation
wouldn't lead to accidental deletions.
Hence, I think we should adjust the comment to maintain its original
point, and extend it to include backup history files. A possible
revision could be (very simple):
* Truncation is essentially harmless, because we skip names of length
* longer than the length of backup history file. (In principle, one
* could use a 1000-character additional_ext and get trouble.)
Regarding the TAP test, it checks that the --clean-backup-history does
indeed remove backup history files. However, it doesn't check that
this doesn't occur if the option isn't specified. Shouldn't we test
for the latter scenario as well?
+sub get_walfiles
+{
<snip..>
+
+create_files(get_walfiles(@walfiles_with_gz));
The new function get_walfiels() just increases the line count without
cutting any lines. The following changes are sufficient and easier to
read (at least for me).
open my $file, '>', "$tempdir/$fn->{name}";
foreach my $fn (map {$_->{name}} @walfiles_with_gz)
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 2023-06-22 16:47, Kyotaro Horiguchi wrote:
Thanks for your review!
0002:
+ * Check file name. + * + * We skip files which are not WAL file or partial WAL file.There's no need to spend this many lines describing this, and it's
suggested to avoid restating what the code does. I think a comment
might not be necessary. But if we decide to have one, it could look
like this:/* we're only removing specific types of files */
As you mentioned, this comment is restatement of the codes.
Removed the comment.
Other than that, it looks good to me.
0003: + <para> + Remove backup history files.I might be overthinking it, but I'd phrase it as, "Remove backup
history files *as well*.". (The --help message describes the same
thing using "including".)
Agreed.
+ For details about backup history file, please refer to the
<xref linkend="backup-base-backup"/>.We usually write this as simple as "See <xref...> for details (of the
backup history files)" or "Refer to <xref..> for more information
(about the backup history files)." or such like... (I think)
Agreed. I used the former one.
+bool cleanBackupHistory = false; /* remove files including + * backup history files */The indentation appears to be inconsistent.
Modified.
- * Truncation is essentially harmless, because we skip names of - * length other than XLOG_FNAME_LEN. (In principle, one could use - * a 1000-character additional_ext and get trouble.) + * Truncation is essentially harmless, because we check the file + * format including the length immediately after this. + * (In principle, one could use a 1000-character additional_ext + * and get trouble.) */ strlcpy(walfile, xlde->d_name, MAXPGPATH); TrimExtension(walfile, additional_ext);The revised comment seems to drift from the original point. Except for
a potential exception by a too-long addition_ext, the original comment
asserted that the name truncation was safe since it wouldn't affect
the files we're removing. In other words, it asserted that the
filenames to be removed won't be truncated and any actual truncation
wouldn't lead to accidental deletions.Hence, I think we should adjust the comment to maintain its original
point, and extend it to include backup history files. A possible
revision could be (very simple):* Truncation is essentially harmless, because we skip names of
length
* longer than the length of backup history file. (In principle, one
* could use a 1000-character additional_ext and get trouble.)
This is true, but we do stricter check for preventing accidental
deletion at the below code than just skipping "names of length longer
than the length of backup history file".
| if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile)
&&
| !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
| continue;
How about something like this?
| Truncation is essentially harmless, because we skip files whose format
is different from WAL files and backup history files.
Regarding the TAP test, it checks that the --clean-backup-history does
indeed remove backup history files. However, it doesn't check that
this doesn't occur if the option isn't specified. Shouldn't we test
for the latter scenario as well?
Agreed.
Added a backup history file to @walfiles_with_gz.
+sub get_walfiles +{ <snip..> + +create_files(get_walfiles(@walfiles_with_gz));The new function get_walfiels() just increases the line count without
cutting any lines. The following changes are sufficient and easier to
read (at least for me).open my $file, '>', "$tempdir/$fn->{name}";
foreach my $fn (map {$_->{name}} @walfiles_with_gz)
Agreed.
Remove get_walfiles() and added some changes as above.
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:
v12-0001-Introduce-pg_archivecleanup-into-getopt_long.patchtext/x-diff; name=v12-0001-Introduce-pg_archivecleanup-into-getopt_long.patchDownload
From f9487f831436ff097595f891e5f4e796fb9943a4 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Wed, 21 Jun 2023 21:42:08 +0900
Subject: [PATCH v12 1/3] Introduce pg_archivecleanup into getopt_long
This patch is a preliminary step to add an easy-to-understand option
to delete backup history files, but it also adds long options to
the existing options.
Reviewed-by: Michael Paquier, Kyotaro Horiguchi, Fujii Masao
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 5 ++++-
src/bin/pg_archivecleanup/pg_archivecleanup.c | 22 +++++++++++++------
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 635e7c7685..09991c2fcd 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -95,6 +95,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-d</option></term>
+ <term><option>--debug</option></term>
<listitem>
<para>
Print lots of debug logging output on <filename>stderr</filename>.
@@ -104,6 +105,7 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<varlistentry>
<term><option>-n</option></term>
+ <term><option>--dry-run</option></term>
<listitem>
<para>
Print the names of the files that would have been removed on <filename>stdout</filename> (performs a dry run).
@@ -122,7 +124,8 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
</varlistentry>
<varlistentry>
- <term><option>-x</option> <replaceable>extension</replaceable></term>
+ <term><option>-x <replaceable class="parameter">extension</replaceable></option></term>
+ <term><option>--strip-extension=<replaceable class="parameter">extension</replaceable></option></term>
<listitem>
<para>
Provide an extension
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 7726d05149..fc0dca9856 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -17,7 +17,7 @@
#include "access/xlog_internal.h"
#include "common/logging.h"
-#include "pg_getopt.h"
+#include "getopt_long.h"
const char *progname;
@@ -252,11 +252,13 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
- printf(_(" -d generate debug output (verbose mode)\n"));
- printf(_(" -n dry run, show the names of the files that would be removed\n"));
- printf(_(" -V, --version output version information, then exit\n"));
- printf(_(" -x EXT clean up files if they have this extension\n"));
- printf(_(" -?, --help show this help, then exit\n"));
+ printf(_(" -d, --debug generate debug output (verbose mode)\n"));
+ printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
+ " removed\n"));
+ printf(_(" -V, --version output version information, then exit\n"));
+ printf(_(" -x, --strip-extension=EXT strip this extention before identifying files for\n"
+ " clean up\n"));
+ printf(_(" -?, --help show this help, then exit\n"));
printf(_("\n"
"For use as archive_cleanup_command in postgresql.conf:\n"
" archive_cleanup_command = 'pg_archivecleanup [OPTION]... ARCHIVELOCATION %%r'\n"
@@ -274,6 +276,12 @@ usage(void)
int
main(int argc, char **argv)
{
+ static struct option long_options[] = {
+ {"debug", no_argument, NULL, 'd'},
+ {"dry-run", no_argument, NULL, 'n'},
+ {"strip-extension", required_argument, NULL, 'x'},
+ {NULL, 0, NULL, 0}
+ };
int c;
pg_logging_init(argv[0]);
@@ -294,7 +302,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt(argc, argv, "dnx:")) != -1)
+ while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
{
switch (c)
{
--
2.39.2
v12-0002-Preliminary-refactoring-for-a-subsequent-patch.patchtext/x-diff; name=v12-0002-Preliminary-refactoring-for-a-subsequent-patch.patchDownload
From caf1889d4fc955f454b98d0b64d17c89e6732490 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Fri, 23 Jun 2023 15:11:25 +0900
Subject: [PATCH v12 2/3] Preliminary refactoring for a subsequent patch
This is a preparatory patch that doesn't introduce any functional
change. Instead, this carries out preliminary refactoring with the
goal of reducing the overall nesting level. This helps to prevent the
forthcoming patch from further deepening the nesting level.
Reviewed-by: Michael Paquier, Kyotaro Horiguchi, Fujii Masao
---
src/bin/pg_archivecleanup/pg_archivecleanup.c | 135 +++++++++---------
1 file changed, 67 insertions(+), 68 deletions(-)
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index fc0dca9856..8d185665ab 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -93,76 +93,75 @@ CleanupPriorWALFiles(void)
struct dirent *xlde;
char walfile[MAXPGPATH];
- if ((xldir = opendir(archiveLocation)) != NULL)
- {
- while (errno = 0, (xlde = readdir(xldir)) != NULL)
- {
- /*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
- */
- strlcpy(walfile, xlde->d_name, MAXPGPATH);
- TrimExtension(walfile, additional_ext);
-
- /*
- * We ignore the timeline part of the XLOG segment identifiers in
- * deciding whether a segment is still needed. This ensures that
- * we won't prematurely remove a segment from a parent timeline.
- * We could probably be a little more proactive about removing
- * segments of non-parent timelines, but that would be a whole lot
- * more complicated.
- *
- * We use the alphanumeric sorting property of the filenames to
- * decide which ones are earlier than the exclusiveCleanupFileName
- * file. Note that this means files are not removed in the order
- * they were originally written, in case this worries you.
- */
- if ((IsXLogFileName(walfile) || IsPartialXLogFileName(walfile)) &&
- strcmp(walfile + 8, exclusiveCleanupFileName + 8) < 0)
- {
- char WALFilePath[MAXPGPATH * 2]; /* the file path
- * including archive */
-
- /*
- * Use the original file name again now, including any
- * extension that might have been chopped off before testing
- * the sequence.
- */
- snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
- archiveLocation, xlde->d_name);
-
- if (dryrun)
- {
- /*
- * Prints the name of the file to be removed and skips the
- * actual removal. The regular printout is so that the
- * user can pipe the output into some other program.
- */
- printf("%s\n", WALFilePath);
- pg_log_debug("file \"%s\" would be removed", WALFilePath);
- continue;
- }
-
- pg_log_debug("removing file \"%s\"", WALFilePath);
-
- rc = unlink(WALFilePath);
- if (rc != 0)
- pg_fatal("could not remove file \"%s\": %m",
- WALFilePath);
- }
- }
-
- if (errno)
- pg_fatal("could not read archive location \"%s\": %m",
- archiveLocation);
- if (closedir(xldir))
- pg_fatal("could not close archive location \"%s\": %m",
- archiveLocation);
- }
- else
+ if ((xldir = opendir(archiveLocation)) == NULL)
pg_fatal("could not open archive location \"%s\": %m",
archiveLocation);
+
+ while (errno = 0, (xlde = readdir(xldir)) != NULL)
+ {
+ char WALFilePath[MAXPGPATH * 2]; /* the file path
+ * including archive */
+ /*
+ * Truncation is essentially harmless, because we skip names of
+ * length other than XLOG_FNAME_LEN. (In principle, one could use
+ * a 1000-character additional_ext and get trouble.)
+ */
+ strlcpy(walfile, xlde->d_name, MAXPGPATH);
+ TrimExtension(walfile, additional_ext);
+
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ continue;
+
+ /*
+ * We ignore the timeline part of the XLOG segment identifiers in
+ * deciding whether a segment is still needed. This ensures that
+ * we won't prematurely remove a segment from a parent timeline.
+ * We could probably be a little more proactive about removing
+ * segments of non-parent timelines, but that would be a whole lot
+ * more complicated.
+ *
+ * We use the alphanumeric sorting property of the filenames to
+ * decide which ones are earlier than the exclusiveCleanupFileName
+ * file. Note that this means files are not removed in the order
+ * they were originally written, in case this worries you.
+ */
+ if (strcmp(walfile + 8, exclusiveCleanupFileName + 8) >= 0)
+ continue;
+
+ /*
+ * Use the original file name again now, including any
+ * extension that might have been chopped off before testing
+ * the sequence.
+ */
+ snprintf(WALFilePath, sizeof(WALFilePath), "%s/%s",
+ archiveLocation, xlde->d_name);
+
+ if (dryrun)
+ {
+ /*
+ * Prints the name of the file to be removed and skips the
+ * actual removal. The regular printout is so that the
+ * user can pipe the output into some other program.
+ */
+ printf("%s\n", WALFilePath);
+ pg_log_debug("file \"%s\" would be removed", WALFilePath);
+ continue;
+ }
+
+ pg_log_debug("removing file \"%s\"", WALFilePath);
+
+ rc = unlink(WALFilePath);
+ if (rc != 0)
+ pg_fatal("could not remove file \"%s\": %m",
+ WALFilePath);
+ }
+
+ if (errno)
+ pg_fatal("could not read archive location \"%s\": %m",
+ archiveLocation);
+ if (closedir(xldir))
+ pg_fatal("could not close archive location \"%s\": %m",
+ archiveLocation);
}
/*
--
2.39.2
v12-0003-Allow-pg_archivecleanup-to-remove-backup-history.patchtext/x-diff; name=v12-0003-Allow-pg_archivecleanup-to-remove-backup-history.patchDownload
From b8df187e3d988531c74a005f0630147aa78e63e6 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Fri, 23 Jun 2023 17:07:10 +0900
Subject: [PATCH v12 3/3] Allow pg_archivecleanup to remove backup history
files
Backup history files are just few bytes, but it can be noisy for the
eye to see a large accumulation of history files mixed with the WAL
segments.
This patch adds a new option to remove files including backup
history files older oldestkeptwalfile.
Reviewed-by: Michael Paquier, Kyotaro Horiguchi, Fujii Masao
---
doc/src/sgml/ref/pgarchivecleanup.sgml | 11 +++
src/bin/pg_archivecleanup/pg_archivecleanup.c | 19 +++--
.../t/010_pg_archivecleanup.pl | 70 ++++++++++++-------
3 files changed, 70 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/ref/pgarchivecleanup.sgml b/doc/src/sgml/ref/pgarchivecleanup.sgml
index 09991c2fcd..9accae8b12 100644
--- a/doc/src/sgml/ref/pgarchivecleanup.sgml
+++ b/doc/src/sgml/ref/pgarchivecleanup.sgml
@@ -93,6 +93,17 @@ pg_archivecleanup: removing file "archive/00000001000000370000000E"
<variablelist>
+ <varlistentry>
+ <term><option>-b</option></term>
+ <term><option>--clean-backup-history</option></term>
+ <listitem>
+ <para>
+ Remove backup history files as well.
+ See <xref linkend="backup-base-backup"/> for details of the backup history files.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><option>-d</option></term>
<term><option>--debug</option></term>
diff --git a/src/bin/pg_archivecleanup/pg_archivecleanup.c b/src/bin/pg_archivecleanup/pg_archivecleanup.c
index 8d185665ab..5774a048e7 100644
--- a/src/bin/pg_archivecleanup/pg_archivecleanup.c
+++ b/src/bin/pg_archivecleanup/pg_archivecleanup.c
@@ -23,6 +23,8 @@ const char *progname;
/* Options and defaults */
bool dryrun = false; /* are we performing a dry-run operation? */
+bool cleanBackupHistory = false; /* remove files including
+ * backup history files */
char *additional_ext = NULL; /* Extension to remove from filenames */
char *archiveLocation; /* where to find the archive? */
@@ -102,14 +104,16 @@ CleanupPriorWALFiles(void)
char WALFilePath[MAXPGPATH * 2]; /* the file path
* including archive */
/*
- * Truncation is essentially harmless, because we skip names of
- * length other than XLOG_FNAME_LEN. (In principle, one could use
- * a 1000-character additional_ext and get trouble.)
+ * Truncation is essentially harmless, because we skip files whose
+ * format is different from WAL files and backup history files.
+ * (In principle, one could use a 1000-character additional_ext and
+ * get trouble.)
*/
strlcpy(walfile, xlde->d_name, MAXPGPATH);
TrimExtension(walfile, additional_ext);
- if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile))
+ if (!IsXLogFileName(walfile) && !IsPartialXLogFileName(walfile) &&
+ !(cleanBackupHistory && IsBackupHistoryFileName(walfile)))
continue;
/*
@@ -251,6 +255,7 @@ usage(void)
printf(_("Usage:\n"));
printf(_(" %s [OPTION]... ARCHIVELOCATION OLDESTKEPTWALFILE\n"), progname);
printf(_("\nOptions:\n"));
+ printf(_(" -b, --clean-backup-history clean up files including backup history files\n"));
printf(_(" -d, --debug generate debug output (verbose mode)\n"));
printf(_(" -n, --dry-run dry run, show the names of the files that would be\n"
" removed\n"));
@@ -276,6 +281,7 @@ int
main(int argc, char **argv)
{
static struct option long_options[] = {
+ {"clean-backup-history", no_argument, NULL, 'b'},
{"debug", no_argument, NULL, 'd'},
{"dry-run", no_argument, NULL, 'n'},
{"strip-extension", required_argument, NULL, 'x'},
@@ -301,10 +307,13 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "dnx:", long_options, NULL)) != -1)
+ while ((c = getopt_long(argc, argv, "bdnx:", long_options, NULL)) != -1)
{
switch (c)
{
+ case 'b': /* Remove backup history files as well */
+ cleanBackupHistory = true;
+ break;
case 'd': /* Debug mode */
pg_logging_increase_verbosity();
break;
diff --git a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
index cc3386d146..2f1964a218 100644
--- a/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
+++ b/src/bin/pg_archivecleanup/t/010_pg_archivecleanup.pl
@@ -12,22 +12,34 @@ program_options_handling_ok('pg_archivecleanup');
my $tempdir = PostgreSQL::Test::Utils::tempdir;
-my @walfiles = (
- '00000001000000370000000C.gz', '00000001000000370000000D',
- '00000001000000370000000E', '00000001000000370000000F.partial',);
+my @walfiles_with_gz = (
+ { name => '00000001000000370000000C.gz', present => 0 },
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.backup', present => 1 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
+
+my @walfiles_for_clean_backup_history = (
+ { name => '00000001000000370000000D', present => 0 },
+ { name => '00000001000000370000000D.00000028.backup', present => 0 },
+ { name => '00000001000000370000000E', present => 1 },
+ { name => '00000001000000370000000F.partial', present => 1 },
+ { name => 'unrelated_file', present => 1 });
sub create_files
{
- foreach my $fn (@walfiles, 'unrelated_file')
+ foreach my $fn (map {$_->{name}} @_)
{
open my $file, '>', "$tempdir/$fn";
+
print $file 'CONTENT';
close $file;
}
return;
}
-create_files();
+create_files(@walfiles_with_gz);
command_fails_like(
['pg_archivecleanup'],
@@ -59,14 +71,14 @@ command_fails_like(
my $stderr;
my $result =
IPC::Run::run [ 'pg_archivecleanup', '-d', '-n', $tempdir,
- $walfiles[2] ],
+ $walfiles_with_gz[3]->{name} ],
'2>', \$stderr;
ok($result, "pg_archivecleanup dry run: exit code 0");
like(
$stderr,
- qr/$walfiles[1].*would be removed/,
+ qr/$walfiles_with_gz[1]->{name}.*would be removed/,
"pg_archivecleanup dry run: matches");
- foreach my $fn (@walfiles)
+ foreach my $fn (map {$_->{name}} @walfiles_with_gz)
{
ok(-f "$tempdir/$fn", "$fn not removed");
}
@@ -76,32 +88,40 @@ sub run_check
{
local $Test::Builder::Level = $Test::Builder::Level + 1;
- my ($suffix, $test_name) = @_;
+ my ($testdata, $oldestkeptwalfile, $test_name, @options) = @_;
- create_files();
+ create_files(@$testdata);
command_ok(
[
- 'pg_archivecleanup', '-x', '.gz', $tempdir,
- $walfiles[2] . $suffix
+ 'pg_archivecleanup', @options, $tempdir,
+ $oldestkeptwalfile
],
"$test_name: runs");
- ok(!-f "$tempdir/$walfiles[0]",
- "$test_name: first older WAL file was cleaned up");
- ok(!-f "$tempdir/$walfiles[1]",
- "$test_name: second older WAL file was cleaned up");
- ok(-f "$tempdir/$walfiles[2]",
- "$test_name: restartfile was not cleaned up");
- ok(-f "$tempdir/$walfiles[3]",
- "$test_name: newer WAL file was not cleaned up");
- ok(-f "$tempdir/unrelated_file",
- "$test_name: unrelated file was not cleaned up");
+ for my $walpair (@$testdata)
+ {
+ if ($walpair->{present})
+ {
+ ok(-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was not cleaned up");
+ }
+ else
+ {
+ ok(!-f "$tempdir/$walpair->{name}",
+ "$test_name:$walpair->{name} was cleaned up");
+ }
+ }
return;
}
-run_check('', 'pg_archivecleanup');
-run_check('.partial', 'pg_archivecleanup with .partial file');
-run_check('.00000020.backup', 'pg_archivecleanup with .backup file');
+run_check(\@walfiles_with_gz, '00000001000000370000000E',
+ 'pg_archivecleanup', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.partial',
+ 'pg_archivecleanup with .partial file', '-x.gz');
+run_check(\@walfiles_with_gz, '00000001000000370000000E.00000020.backup',
+ 'pg_archivecleanup with .backup file', '-x.gz');
+run_check(\@walfiles_for_clean_backup_history, '00000001000000370000000E',
+ 'pg_archivecleanup with --clean-backup-history', '-b');
done_testing();
--
2.39.2
On Fri, Jun 23, 2023 at 05:37:09PM +0900, torikoshia wrote:
On 2023-06-22 16:47, Kyotaro Horiguchi wrote:
Thanks for your review!
I have begun cleaning up my board, and applied 0001 for the moment.
--
Michael
On Fri, Jun 30, 2023 at 03:48:43PM +0900, Michael Paquier wrote:
I have begun cleaning up my board, and applied 0001 for the moment.
And a few weeks later.. I have come around this thread and applied
0002 and 0003.
The flow of 0002 was straight-forward. My main issue was in 0003,
actually, where the TAP tests were kind of confusing as written:
- There was no cleanup of the files still present after a single
command check, which could easily mess up the tests.
- The --dry-run case was using the list of WAL files for the extension
pattern checks, hardcoding names based on the position of its array.
I have switched that to use a third list of files, instead.
The result looked OK and that can be extended easily for more
patterns or more commands, so applied 0003 after doing these
adjustments, coupled with a pgperltidy run, a pgperlcritic check and
an indentation.
--
Michael
On 2023-07-19 13:58, Michael Paquier wrote:
On Fri, Jun 30, 2023 at 03:48:43PM +0900, Michael Paquier wrote:
I have begun cleaning up my board, and applied 0001 for the moment.
And a few weeks later.. I have come around this thread and applied
0002 and 0003.The flow of 0002 was straight-forward. My main issue was in 0003,
actually, where the TAP tests were kind of confusing as written:
- There was no cleanup of the files still present after a single
command check, which could easily mess up the tests.
- The --dry-run case was using the list of WAL files for the extension
pattern checks, hardcoding names based on the position of its array.
I have switched that to use a third list of files, instead.The result looked OK and that can be extended easily for more
patterns or more commands, so applied 0003 after doing these
adjustments, coupled with a pgperltidy run, a pgperlcritic check and
an indentation.
--
Michael
Thanks for the reviewing and applying the patches!
--
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION