Is it correct to say, "invalid data in file \"%s\"", BACKUP_LABEL_FILE in do_pg_backup_stop?
Hi,
After the commit [1]commit 39969e2a1e4d7f5a37f3ef37d53bbfe171e7d77a Author: Stephen Frost <sfrost@snowman.net> Date: Wed Apr 6 14:41:03 2022 -0400, is it correct to say errmsg("invalid data in file
\"%s\"", BACKUP_LABEL_FILE))); in do_pg_backup_stop() when we hold the
contents in backend global memory, not actually reading from backup_label
file? However, it is correct to say that in read_backup_label.
IMO, we can either say "invalid backup_label contents found" or we can be
more descriptive and say "invalid "START WAL LOCATION" line found in
backup_label content" and "invalid "BACKUP FROM" line found in
backup_label content" and so on.
Thoughts?
[1]: commit 39969e2a1e4d7f5a37f3ef37d53bbfe171e7d77a Author: Stephen Frost <sfrost@snowman.net> Date: Wed Apr 6 14:41:03 2022 -0400
commit 39969e2a1e4d7f5a37f3ef37d53bbfe171e7d77a
Author: Stephen Frost <sfrost@snowman.net>
Date: Wed Apr 6 14:41:03 2022 -0400
Remove exclusive backup mode
errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
Regards,
Bharath Rupireddy.
At Wed, 20 Jul 2022 17:09:09 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hi,
After the commit [1], is it correct to say errmsg("invalid data in file
\"%s\"", BACKUP_LABEL_FILE))); in do_pg_backup_stop() when we hold the
contents in backend global memory, not actually reading from backup_label
file? However, it is correct to say that in read_backup_label.IMO, we can either say "invalid backup_label contents found" or we can be
more descriptive and say "invalid "START WAL LOCATION" line found in
backup_label content" and "invalid "BACKUP FROM" line found in
backup_label content" and so on.Thoughts?
Previously there the case the "char *labelfile" is loaded from a file,
but currently it is alwasy a string build on the process. In that
sense, nowadays it is a kind of internal error, which I think is not
supposed to be exposed to users.
So I think we can leave the code alone to avoid back-patching
obstacles. But if we decided to change the code around, I'd like to
change the string into a C struct, so that we don't need to parse it.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Thu, Jul 21, 2022 at 2:33 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
At Wed, 20 Jul 2022 17:09:09 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hi,
After the commit [1], is it correct to say errmsg("invalid data in file
\"%s\"", BACKUP_LABEL_FILE))); in do_pg_backup_stop() when we hold the
contents in backend global memory, not actually reading from backup_label
file? However, it is correct to say that in read_backup_label.IMO, we can either say "invalid backup_label contents found" or we can be
more descriptive and say "invalid "START WAL LOCATION" line found in
backup_label content" and "invalid "BACKUP FROM" line found in
backup_label content" and so on.Thoughts?
Previously there the case the "char *labelfile" is loaded from a file,
but currently it is alwasy a string build on the process. In that
sense, nowadays it is a kind of internal error, which I think is not
supposed to be exposed to users.So I think we can leave the code alone to avoid back-patching
obstacles. But if we decided to change the code around, I'd like to
change the string into a C struct, so that we don't need to parse it.
Hm. I think we must take this opportunity to clean it up. You are
right, we don't need to parse the label file contents (just like we
used to do previously after reading it from the file) in
do_pg_backup_stop(), instead we can just pass a structure. Also,
do_pg_backup_stop() isn't modifying any labelfile contents, but using
startxlogfilename, startpoint and backupfrom from the labelfile
contents. I think this information can easily be passed as a single
structure. In fact, I might think a bit more here and wrap label_file,
tblspc_map_file to a single structure something like below and pass it
across the functions.
typedef struct BackupState
{
StringInfo label_file;
StringInfo tblspc_map_file;
char startxlogfilename[MAXFNAMELEN];
XLogRecPtr startpoint;
char backupfrom[20];
} BackupState;
This way, the code is more readable, structured and we can remove 2
sscanf() calls, 2 "invalid data in file" errors, 1 strchr() call, 1
strstr() call. Only thing is that it creates code diff from the
previous PG versions which is fine IMO. If okay, I'm happy to prepare
a patch.
Thoughts?
Regards,
Bharath Rupireddy.
At Mon, 25 Jul 2022 14:21:38 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hm. I think we must take this opportunity to clean it up. You are
right, we don't need to parse the label file contents (just like we
used to do previously after reading it from the file) in
do_pg_backup_stop(), instead we can just pass a structure. Also,
do_pg_backup_stop() isn't modifying any labelfile contents, but using
startxlogfilename, startpoint and backupfrom from the labelfile
contents. I think this information can easily be passed as a single
structure. In fact, I might think a bit more here and wrap label_file,
tblspc_map_file to a single structure something like below and pass it
across the functions.typedef struct BackupState
{
StringInfo label_file;
StringInfo tblspc_map_file;
char startxlogfilename[MAXFNAMELEN];
XLogRecPtr startpoint;
char backupfrom[20];
} BackupState;This way, the code is more readable, structured and we can remove 2
sscanf() calls, 2 "invalid data in file" errors, 1 strchr() call, 1
strstr() call. Only thing is that it creates code diff from the
previous PG versions which is fine IMO. If okay, I'm happy to prepare
a patch.Thoughts?
It is more or less what was in my mind, but it seems that we don't
need StringInfo there, or should avoid it to signal the strings are
not editable.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On 7/25/22 22:49, Kyotaro Horiguchi wrote:
At Mon, 25 Jul 2022 14:21:38 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hm. I think we must take this opportunity to clean it up. You are
right, we don't need to parse the label file contents (just like we
used to do previously after reading it from the file) in
do_pg_backup_stop(), instead we can just pass a structure. Also,
do_pg_backup_stop() isn't modifying any labelfile contents, but using
startxlogfilename, startpoint and backupfrom from the labelfile
contents. I think this information can easily be passed as a single
structure. In fact, I might think a bit more here and wrap label_file,
tblspc_map_file to a single structure something like below and pass it
across the functions.typedef struct BackupState
{
StringInfo label_file;
StringInfo tblspc_map_file;
char startxlogfilename[MAXFNAMELEN];
XLogRecPtr startpoint;
char backupfrom[20];
} BackupState;This way, the code is more readable, structured and we can remove 2
sscanf() calls, 2 "invalid data in file" errors, 1 strchr() call, 1
strstr() call. Only thing is that it creates code diff from the
previous PG versions which is fine IMO. If okay, I'm happy to prepare
a patch.Thoughts?
It is more or less what was in my mind, but it seems that we don't
need StringInfo there, or should avoid it to signal the strings are
not editable.
I would prefer to have all the components of backup_label stored
separately and then generate backup_label from them in pg_backup_stop().
For PG16 I am planning to add some fields to backup_label that are not
known when pg_backup_start() is called, e.g. min recovery time.
Regards,
-David
On Tue, Jul 26, 2022 at 5:22 PM David Steele <david@pgmasters.net> wrote:
On 7/25/22 22:49, Kyotaro Horiguchi wrote:
At Mon, 25 Jul 2022 14:21:38 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hm. I think we must take this opportunity to clean it up. You are
right, we don't need to parse the label file contents (just like we
used to do previously after reading it from the file) in
do_pg_backup_stop(), instead we can just pass a structure. Also,
do_pg_backup_stop() isn't modifying any labelfile contents, but using
startxlogfilename, startpoint and backupfrom from the labelfile
contents. I think this information can easily be passed as a single
structure. In fact, I might think a bit more here and wrap label_file,
tblspc_map_file to a single structure something like below and pass it
across the functions.typedef struct BackupState
{
StringInfo label_file;
StringInfo tblspc_map_file;
char startxlogfilename[MAXFNAMELEN];
XLogRecPtr startpoint;
char backupfrom[20];
} BackupState;This way, the code is more readable, structured and we can remove 2
sscanf() calls, 2 "invalid data in file" errors, 1 strchr() call, 1
strstr() call. Only thing is that it creates code diff from the
previous PG versions which is fine IMO. If okay, I'm happy to prepare
a patch.Thoughts?
It is more or less what was in my mind, but it seems that we don't
need StringInfo there, or should avoid it to signal the strings are
not editable.I would prefer to have all the components of backup_label stored
separately and then generate backup_label from them in pg_backup_stop().
+1, because pg_backup_stop is the one that's returning backup_label
contents, so it does make sense for it to prepare it once and for all
and return.
For PG16 I am planning to add some fields to backup_label that are not
known when pg_backup_start() is called, e.g. min recovery time.
Can you please point to your patch that does above?
Yes, right now, backup_label or tablespace_map contents are being
filled in by pg_backup_start and are never changed again. But if your
above proposal is for fixing some issue, then it would make sense for
us to carry all the info in a structure to pg_backup_stop and then let
it prepare the backup_label and tablespace_map contents.
If the approach is okay for the hackers, I would like to spend time on it.
Regards,
Bharath Rupireddy.
On 7/26/22 07:59, Bharath Rupireddy wrote:
On Tue, Jul 26, 2022 at 5:22 PM David Steele <david@pgmasters.net> wrote:
On 7/25/22 22:49, Kyotaro Horiguchi wrote:
At Mon, 25 Jul 2022 14:21:38 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
Hm. I think we must take this opportunity to clean it up. You are
right, we don't need to parse the label file contents (just like we
used to do previously after reading it from the file) in
do_pg_backup_stop(), instead we can just pass a structure. Also,
do_pg_backup_stop() isn't modifying any labelfile contents, but using
startxlogfilename, startpoint and backupfrom from the labelfile
contents. I think this information can easily be passed as a single
structure. In fact, I might think a bit more here and wrap label_file,
tblspc_map_file to a single structure something like below and pass it
across the functions.typedef struct BackupState
{
StringInfo label_file;
StringInfo tblspc_map_file;
char startxlogfilename[MAXFNAMELEN];
XLogRecPtr startpoint;
char backupfrom[20];
} BackupState;This way, the code is more readable, structured and we can remove 2
sscanf() calls, 2 "invalid data in file" errors, 1 strchr() call, 1
strstr() call. Only thing is that it creates code diff from the
previous PG versions which is fine IMO. If okay, I'm happy to prepare
a patch.Thoughts?
It is more or less what was in my mind, but it seems that we don't
need StringInfo there, or should avoid it to signal the strings are
not editable.I would prefer to have all the components of backup_label stored
separately and then generate backup_label from them in pg_backup_stop().+1, because pg_backup_stop is the one that's returning backup_label
contents, so it does make sense for it to prepare it once and for all
and return.For PG16 I am planning to add some fields to backup_label that are not
known when pg_backup_start() is called, e.g. min recovery time.Can you please point to your patch that does above?
Currently it is a plan, not a patch. So there is nothing to show yet.
Yes, right now, backup_label or tablespace_map contents are being
filled in by pg_backup_start and are never changed again. But if your
above proposal is for fixing some issue, then it would make sense for
us to carry all the info in a structure to pg_backup_stop and then let
it prepare the backup_label and tablespace_map contents.
I think this makes sense even if I don't get these changes into PG16.
If the approach is okay for the hackers, I would like to spend time on it.
+1 from me.
Regards,
-David
On Tue, Jul 26, 2022 at 5:50 PM David Steele <david@pgmasters.net> wrote:
I would prefer to have all the components of backup_label stored
separately and then generate backup_label from them in pg_backup_stop().+1, because pg_backup_stop is the one that's returning backup_label
contents, so it does make sense for it to prepare it once and for all
and return.For PG16 I am planning to add some fields to backup_label that are not
known when pg_backup_start() is called, e.g. min recovery time.Can you please point to your patch that does above?
Currently it is a plan, not a patch. So there is nothing to show yet.
Yes, right now, backup_label or tablespace_map contents are being
filled in by pg_backup_start and are never changed again. But if your
above proposal is for fixing some issue, then it would make sense for
us to carry all the info in a structure to pg_backup_stop and then let
it prepare the backup_label and tablespace_map contents.I think this makes sense even if I don't get these changes into PG16.
If the approach is okay for the hackers, I would like to spend time on it.
+1 from me.
Here comes the v1 patch. This patch tries to refactor backup related
code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now in pg_backup_stop, no
error checking for invalid parsing.
3) backup_label and history file contents have most of the things in
common, they can now be created within a single function.
4) makes backup related code extensible and readable.
One downside is that it creates a lot of diff with previous versions.
Please review.
--
Bharath Rupireddy
RDS Open Source Databases: https://aws.amazon.com/
Attachments:
v1-0001-Refactor-backup-related-code.patchapplication/x-patch; name=v1-0001-Refactor-backup-related-code.patchDownload
From 441f4c967187e52cc5e2b98046886c91105213cb Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Fri, 29 Jul 2022 23:57:12 +0000
Subject: [PATCH v1] Refactor backup related code
This patch tries to refactor backup related code, advantages of
doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function
4) makes backup related code extensible and readable
---
src/backend/access/transam/xlog.c | 318 +++++++++++++------------
src/backend/access/transam/xlogfuncs.c | 48 ++--
src/backend/replication/basebackup.c | 31 ++-
src/include/access/xlog.h | 12 +-
src/include/access/xlog_internal.h | 25 ++
5 files changed, 234 insertions(+), 200 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 15ab8d90d4..4024eb6320 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8061,62 +8061,137 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
PendingWalStats.wal_sync++;
}
+/*
+ * Get backup state.
+ */
+BackupState
+get_backup_state(const char *name)
+{
+ BackupState state;
+ Size len;
+
+ state = (BackupState) palloc0(sizeof(BackupStateData));
+ len = strlen(name);
+ state->name = (char *) palloc0(len + 1);
+ memcpy(state->name, name, len);
+ state->backup_label = makeStringInfo();
+ state->tablespace_map = makeStringInfo();
+ state->history_file = makeStringInfo();
+
+ return state;
+}
+
+/*
+ * Free backup state.
+ */
+void
+free_backup_state(BackupState state)
+{
+ Assert(state != NULL);
+
+ pfree(state->name);
+ pfree(state->backup_label->data);
+ pfree(state->tablespace_map->data);
+ pfree(state->history_file->data);
+ pfree(state);
+}
+
+/*
+ * Construct backup_label or history file content strings.
+ */
+void
+create_backup_content_str(BackupState state, bool forhistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+
+ if (forhistoryfile)
+ str = state->history_file;
+ else
+ str = state->backup_label;
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->startxlogfile);
+
+ if (forhistoryfile)
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->stopxlogfile);
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (forhistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
+
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * backup state.
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState state, bool fast, List **tablespaces)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
XLogSegNo _logSegNo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ state->started_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!state->started_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8178,7 +8253,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* the backup taken during recovery is not available for the special
* recovery case described above.
*/
- if (!backup_started_in_recovery)
+ if (!state->started_in_recovery)
RequestXLogSwitch(false);
do
@@ -8213,13 +8288,13 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
- if (backup_started_in_recovery)
+ if (state->started_in_recovery)
{
XLogRecPtr recptr;
@@ -8232,7 +8307,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8264,16 +8339,17 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->startxlogfile, state->starttli, _logSegNo,
+ wal_segment_size);
/*
* Construct tablespace_map file.
@@ -8354,7 +8430,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(state->tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
@@ -8372,25 +8448,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
@@ -8398,13 +8456,6 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8436,48 +8487,39 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates backup
+ * label contents and history file (if required), resets sessionBackupState
+ * and so on. It can optionally wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8518,29 +8560,10 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true && in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8576,7 +8599,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (in_recovery)
{
XLogRecPtr recptr;
@@ -8588,7 +8611,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8600,8 +8623,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8610,14 +8633,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8625,39 +8649,30 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->stopxlogfile, state->stoptli, _logSegNo,
+ wal_segment_size);
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* construct history file contents */
+ create_backup_content_str(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8673,6 +8688,9 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
CleanupBackupHistory();
}
+ /* construct backup_label contents */
+ create_backup_content_str(state, false);
+
/*
* If archiving is enabled, wait for all the required WAL files to be
* archived before returning. If archiving isn't enabled, the required WAL
@@ -8695,15 +8713,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!in_recovery && XLogArchivingActive()) ||
+ (in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8744,13 +8763,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 61e0f4a29c..1c6b9e8b06 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -38,11 +38,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +58,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +69,18 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+ backup_state = get_backup_state(backupidstr);
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ do_pg_backup_start(backup_state, fast, NULL);
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -104,13 +97,11 @@ pg_backup_start(PG_FUNCTION_ARGS)
Datum
pg_backup_stop(PG_FUNCTION_ARGS)
{
-#define PG_STOP_BACKUP_V2_COLS 3
+#define PG_BACKUP_STOP_V2_COLS 3
TupleDesc tupdesc;
- Datum values[PG_STOP_BACKUP_V2_COLS] = {0};
- bool nulls[PG_STOP_BACKUP_V2_COLS] = {0};
-
+ Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
+ bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +118,14 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ free_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c
index 7f85071229..31924d3257 100644
--- a/src/backend/replication/basebackup.c
+++ b/src/backend/replication/basebackup.c
@@ -230,9 +230,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -247,18 +246,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = get_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -315,14 +314,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ create_backup_content_str(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -373,7 +377,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ free_backup_state(backup_state);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index cd674c3c23..202495847c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlog_internal.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -279,11 +280,12 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern BackupState get_backup_state(const char *name);
+extern void free_backup_state(BackupState state);
+extern void create_backup_content_str(BackupState state, bool forhistoryfile);
+extern void do_pg_backup_start(BackupState state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 44291b337b..462520342a 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid)
}
#endif
+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name;
+ XLogRecPtr startpoint; /* backup start WAL location */
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ StringInfo backup_label; /* backup_label file contents. */
+ StringInfo tablespace_map; /* tablespace_map file contents. */
+ StringInfo history_file; /* history file contents. */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupStateData;
+
+typedef BackupStateData *BackupState;
+
/*
* Exported to support xlog switching from checkpointer
*/
--
2.34.1
On Sat, Jul 30, 2022 at 5:37 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
On Tue, Jul 26, 2022 at 5:50 PM David Steele <david@pgmasters.net> wrote:
I would prefer to have all the components of backup_label stored
separately and then generate backup_label from them in pg_backup_stop().+1, because pg_backup_stop is the one that's returning backup_label
contents, so it does make sense for it to prepare it once and for all
and return.For PG16 I am planning to add some fields to backup_label that are not
known when pg_backup_start() is called, e.g. min recovery time.Can you please point to your patch that does above?
Currently it is a plan, not a patch. So there is nothing to show yet.
Yes, right now, backup_label or tablespace_map contents are being
filled in by pg_backup_start and are never changed again. But if your
above proposal is for fixing some issue, then it would make sense for
us to carry all the info in a structure to pg_backup_stop and then let
it prepare the backup_label and tablespace_map contents.I think this makes sense even if I don't get these changes into PG16.
If the approach is okay for the hackers, I would like to spend time on it.
+1 from me.
Here comes the v1 patch. This patch tries to refactor backup related
code, advantages of doing so are following:1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now in pg_backup_stop, no
error checking for invalid parsing.
3) backup_label and history file contents have most of the things in
common, they can now be created within a single function.
4) makes backup related code extensible and readable.One downside is that it creates a lot of diff with previous versions.
Please review.
I added this to current CF - https://commitfest.postgresql.org/39/3808/
--
Bharath Rupireddy
RDS Open Source Databases: https://aws.amazon.com/rds/postgresql/
On Mon, Aug 8, 2022 at 7:20 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
Please review.
I added this to current CF - https://commitfest.postgresql.org/39/3808/
Here's the v2 patch, no change from v1, just rebased because of commit
a8c012869763c711abc9085f54b2a100b60a85fa (Move basebackup code to new
directory src/backend/backup).
--
Bharath Rupireddy
RDS Open Source Databases: https://aws.amazon.com/rds/postgresql/
Attachments:
v2-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v2-0001-Refactor-backup-related-code.patchDownload
From d42641d979ea10b5456c98ee393425a1cf905aca Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Thu, 11 Aug 2022 04:21:54 +0000
Subject: [PATCH v2] Refactor backup related code
This patch tries to refactor backup related code, advantages of
doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function
4) makes backup related code extensible and readable
---
src/backend/access/transam/xlog.c | 318 +++++++++++++------------
src/backend/access/transam/xlogfuncs.c | 48 ++--
src/backend/backup/basebackup.c | 31 ++-
src/include/access/xlog.h | 12 +-
src/include/access/xlog_internal.h | 25 ++
5 files changed, 234 insertions(+), 200 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 9cedd6876f..4dfa9d9738 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8057,62 +8057,137 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
PendingWalStats.wal_sync++;
}
+/*
+ * Get backup state.
+ */
+BackupState
+get_backup_state(const char *name)
+{
+ BackupState state;
+ Size len;
+
+ state = (BackupState) palloc0(sizeof(BackupStateData));
+ len = strlen(name);
+ state->name = (char *) palloc0(len + 1);
+ memcpy(state->name, name, len);
+ state->backup_label = makeStringInfo();
+ state->tablespace_map = makeStringInfo();
+ state->history_file = makeStringInfo();
+
+ return state;
+}
+
+/*
+ * Free backup state.
+ */
+void
+free_backup_state(BackupState state)
+{
+ Assert(state != NULL);
+
+ pfree(state->name);
+ pfree(state->backup_label->data);
+ pfree(state->tablespace_map->data);
+ pfree(state->history_file->data);
+ pfree(state);
+}
+
+/*
+ * Construct backup_label or history file content strings.
+ */
+void
+create_backup_content_str(BackupState state, bool forhistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+
+ if (forhistoryfile)
+ str = state->history_file;
+ else
+ str = state->backup_label;
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->startxlogfile);
+
+ if (forhistoryfile)
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->stopxlogfile);
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (forhistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
+
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * backup state.
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState state, bool fast, List **tablespaces)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
XLogSegNo _logSegNo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ state->started_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!state->started_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8174,7 +8249,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* the backup taken during recovery is not available for the special
* recovery case described above.
*/
- if (!backup_started_in_recovery)
+ if (!state->started_in_recovery)
RequestXLogSwitch(false);
do
@@ -8209,13 +8284,13 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
- if (backup_started_in_recovery)
+ if (state->started_in_recovery)
{
XLogRecPtr recptr;
@@ -8228,7 +8303,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8260,16 +8335,17 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->startxlogfile, state->starttli, _logSegNo,
+ wal_segment_size);
/*
* Construct tablespace_map file.
@@ -8349,32 +8425,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(state->tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
@@ -8382,13 +8440,6 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8420,48 +8471,39 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates backup
+ * label contents and history file (if required), resets sessionBackupState
+ * and so on. It can optionally wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8502,29 +8544,10 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true && in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8560,7 +8583,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (in_recovery)
{
XLogRecPtr recptr;
@@ -8572,7 +8595,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8584,8 +8607,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8594,14 +8617,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8609,39 +8633,30 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->stopxlogfile, state->stoptli, _logSegNo,
+ wal_segment_size);
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* construct history file contents */
+ create_backup_content_str(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8657,6 +8672,9 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
CleanupBackupHistory();
}
+ /* construct backup_label contents */
+ create_backup_content_str(state, false);
+
/*
* If archiving is enabled, wait for all the required WAL files to be
* archived before returning. If archiving isn't enabled, the required WAL
@@ -8679,15 +8697,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!in_recovery && XLogArchivingActive()) ||
+ (in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8728,13 +8747,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 61e0f4a29c..1c6b9e8b06 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -38,11 +38,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +58,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +69,18 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+ backup_state = get_backup_state(backupidstr);
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ do_pg_backup_start(backup_state, fast, NULL);
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -104,13 +97,11 @@ pg_backup_start(PG_FUNCTION_ARGS)
Datum
pg_backup_stop(PG_FUNCTION_ARGS)
{
-#define PG_STOP_BACKUP_V2_COLS 3
+#define PG_BACKUP_STOP_V2_COLS 3
TupleDesc tupdesc;
- Datum values[PG_STOP_BACKUP_V2_COLS] = {0};
- bool nulls[PG_STOP_BACKUP_V2_COLS] = {0};
-
+ Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
+ bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +118,14 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ free_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 715428029b..5fd55f189b 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -230,9 +230,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -247,18 +246,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = get_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -315,14 +314,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ create_backup_content_str(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -373,7 +377,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ free_backup_state(backup_state);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index cd674c3c23..202495847c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlog_internal.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -279,11 +280,12 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern BackupState get_backup_state(const char *name);
+extern void free_backup_state(BackupState state);
+extern void create_backup_content_str(BackupState state, bool forhistoryfile);
+extern void do_pg_backup_start(BackupState state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 44291b337b..462520342a 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid)
}
#endif
+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name;
+ XLogRecPtr startpoint; /* backup start WAL location */
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ StringInfo backup_label; /* backup_label file contents. */
+ StringInfo tablespace_map; /* tablespace_map file contents. */
+ StringInfo history_file; /* history file contents. */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupStateData;
+
+typedef BackupStateData *BackupState;
+
/*
* Exported to support xlog switching from checkpointer
*/
--
2.34.1
On Thu, Aug 11, 2022 at 09:55:13AM +0530, Bharath Rupireddy wrote:
Here's the v2 patch, no change from v1, just rebased because of commit
a8c012869763c711abc9085f54b2a100b60a85fa (Move basebackup code to new
directory src/backend/backup).
I was skimming at this patch, and indeed it is a bit crazy to write
the generate the contents of the backup_label file at backup start,
just to parse them again at backup stop with these extra sscan calls.
-#define PG_STOP_BACKUP_V2_COLS 3
+#define PG_BACKUP_STOP_V2_COLS 3
It seems to me that such changes, while they make sense with the new
naming of the backup start/stop functions are unrelated to what you
are trying to solve primarily here. This justifies a separate
cleanup, but I am perhaps overly-pedantic here :)
--
Michael
On Mon, Sep 12, 2022 at 1:12 PM Michael Paquier <michael@paquier.xyz> wrote:
On Thu, Aug 11, 2022 at 09:55:13AM +0530, Bharath Rupireddy wrote:
Here's the v2 patch, no change from v1, just rebased because of commit
a8c012869763c711abc9085f54b2a100b60a85fa (Move basebackup code to new
directory src/backend/backup).I was skimming at this patch, and indeed it is a bit crazy to write
the generate the contents of the backup_label file at backup start,
just to parse them again at backup stop with these extra sscan calls.
Thanks for taking a look at the patch.
-#define PG_STOP_BACKUP_V2_COLS 3
+#define PG_BACKUP_STOP_V2_COLS 3
It seems to me that such changes, while they make sense with the new
naming of the backup start/stop functions are unrelated to what you
are trying to solve primarily here. This justifies a separate
cleanup, but I am perhaps overly-pedantic here :)
I've posted a separate patch [1]/messages/by-id/CALj2ACXjvC28ppeDTCrfaSyHga0ggP5nRLJbsjx=7N-74UT4QA@mail.gmail.com to adjust the macro name alone.
Please review the attached v3 patch after removing the above macro changes.
[1]: /messages/by-id/CALj2ACXjvC28ppeDTCrfaSyHga0ggP5nRLJbsjx=7N-74UT4QA@mail.gmail.com
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v3-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v3-0001-Refactor-backup-related-code.patchDownload
From 8d39f4d7547cd5d6bde6bbdfac7331b3630e8120 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 12 Sep 2022 11:22:31 +0000
Subject: [PATCH v3] Refactor backup related code
This patch tries to refactor backup related code, advantages of
doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function
4) makes backup related code extensible and readable
---
src/backend/access/transam/xlog.c | 318 +++++++++++++------------
src/backend/access/transam/xlogfuncs.c | 42 ++--
src/backend/backup/basebackup.c | 31 ++-
src/include/access/xlog.h | 12 +-
src/include/access/xlog_internal.h | 25 ++
5 files changed, 231 insertions(+), 197 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7a710e6490..3594d1d186 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8059,62 +8059,137 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
PendingWalStats.wal_sync++;
}
+/*
+ * Get backup state.
+ */
+BackupState
+get_backup_state(const char *name)
+{
+ BackupState state;
+ Size len;
+
+ state = (BackupState) palloc0(sizeof(BackupStateData));
+ len = strlen(name);
+ state->name = (char *) palloc0(len + 1);
+ memcpy(state->name, name, len);
+ state->backup_label = makeStringInfo();
+ state->tablespace_map = makeStringInfo();
+ state->history_file = makeStringInfo();
+
+ return state;
+}
+
+/*
+ * Free backup state.
+ */
+void
+free_backup_state(BackupState state)
+{
+ Assert(state != NULL);
+
+ pfree(state->name);
+ pfree(state->backup_label->data);
+ pfree(state->tablespace_map->data);
+ pfree(state->history_file->data);
+ pfree(state);
+}
+
+/*
+ * Construct backup_label or history file content strings.
+ */
+void
+create_backup_content_str(BackupState state, bool forhistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+
+ if (forhistoryfile)
+ str = state->history_file;
+ else
+ str = state->backup_label;
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->startxlogfile);
+
+ if (forhistoryfile)
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->stopxlogfile);
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (forhistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
+
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * backup state.
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState state, bool fast, List **tablespaces)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
XLogSegNo _logSegNo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ state->started_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!state->started_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8176,7 +8251,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* the backup taken during recovery is not available for the special
* recovery case described above.
*/
- if (!backup_started_in_recovery)
+ if (!state->started_in_recovery)
RequestXLogSwitch(false);
do
@@ -8211,13 +8286,13 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
- if (backup_started_in_recovery)
+ if (state->started_in_recovery)
{
XLogRecPtr recptr;
@@ -8230,7 +8305,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8262,16 +8337,17 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->startxlogfile, state->starttli, _logSegNo,
+ wal_segment_size);
/*
* Construct tablespace_map file.
@@ -8351,32 +8427,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(state->tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
@@ -8384,13 +8442,6 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8422,48 +8473,39 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates backup
+ * label contents and history file (if required), resets sessionBackupState
+ * and so on. It can optionally wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8504,29 +8546,10 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true && in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8562,7 +8585,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (in_recovery)
{
XLogRecPtr recptr;
@@ -8574,7 +8597,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8586,8 +8609,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8596,14 +8619,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8611,39 +8635,30 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->stopxlogfile, state->stoptli, _logSegNo,
+ wal_segment_size);
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* construct history file contents */
+ create_backup_content_str(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8659,6 +8674,9 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
CleanupBackupHistory();
}
+ /* construct backup_label contents */
+ create_backup_content_str(state, false);
+
/*
* If archiving is enabled, wait for all the required WAL files to be
* archived before returning. If archiving isn't enabled, the required WAL
@@ -8681,15 +8699,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!in_recovery && XLogArchivingActive()) ||
+ (in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8730,13 +8749,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 9cc757f1af..8f58008fd5 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -38,11 +38,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +58,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +69,18 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+ backup_state = get_backup_state(backupidstr);
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ do_pg_backup_start(backup_state, fast, NULL);
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,9 +101,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_STOP_BACKUP_V2_COLS] = {0};
bool nulls[PG_STOP_BACKUP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +118,14 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ free_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 3bf3aa6faa..61935a182e 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -230,9 +230,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -247,18 +246,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = get_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -315,14 +314,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ create_backup_content_str(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -373,7 +377,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ free_backup_state(backup_state);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index cd674c3c23..202495847c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlog_internal.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -279,11 +280,12 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern BackupState get_backup_state(const char *name);
+extern void free_backup_state(BackupState state);
+extern void create_backup_content_str(BackupState state, bool forhistoryfile);
+extern void do_pg_backup_start(BackupState state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 44291b337b..462520342a 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid)
}
#endif
+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name;
+ XLogRecPtr startpoint; /* backup start WAL location */
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ StringInfo backup_label; /* backup_label file contents. */
+ StringInfo tablespace_map; /* tablespace_map file contents. */
+ StringInfo history_file; /* history file contents. */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupStateData;
+
+typedef BackupStateData *BackupState;
+
/*
* Exported to support xlog switching from checkpointer
*/
--
2.34.1
On Mon, Sep 12, 2022 at 5:09 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
Please review the attached v3 patch after removing the above macro changes.
I'm attaching the v4 patch that's rebased on to the latest HEAD.
Please consider this for review.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v4-0001-Refactor-backup-related-code.patchapplication/x-patch; name=v4-0001-Refactor-backup-related-code.patchDownload
From a6f39ab4391d2a7582f354888c68b908e92653d8 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Wed, 14 Sep 2022 08:50:10 +0000
Subject: [PATCH v4] Refactor backup related code
This patch tries to refactor backup related code, advantages of
doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function
4) makes backup related code extensible and readable
---
src/backend/access/transam/xlog.c | 318 +++++++++++++------------
src/backend/access/transam/xlogfuncs.c | 42 ++--
src/backend/backup/basebackup.c | 31 ++-
src/include/access/xlog.h | 12 +-
src/include/access/xlog_internal.h | 25 ++
5 files changed, 231 insertions(+), 197 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 81d339d57d..d8361272a4 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8233,62 +8233,137 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
PendingWalStats.wal_sync++;
}
+/*
+ * Get backup state.
+ */
+BackupState
+get_backup_state(const char *name)
+{
+ BackupState state;
+ Size len;
+
+ state = (BackupState) palloc0(sizeof(BackupStateData));
+ len = strlen(name);
+ state->name = (char *) palloc0(len + 1);
+ memcpy(state->name, name, len);
+ state->backup_label = makeStringInfo();
+ state->tablespace_map = makeStringInfo();
+ state->history_file = makeStringInfo();
+
+ return state;
+}
+
+/*
+ * Free backup state.
+ */
+void
+free_backup_state(BackupState state)
+{
+ Assert(state != NULL);
+
+ pfree(state->name);
+ pfree(state->backup_label->data);
+ pfree(state->tablespace_map->data);
+ pfree(state->history_file->data);
+ pfree(state);
+}
+
+/*
+ * Construct backup_label or history file content strings.
+ */
+void
+create_backup_content_str(BackupState state, bool forhistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+
+ if (forhistoryfile)
+ str = state->history_file;
+ else
+ str = state->backup_label;
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->startxlogfile);
+
+ if (forhistoryfile)
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->stopxlogfile);
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (forhistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
+
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * backup state.
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState state, bool fast, List **tablespaces)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
XLogSegNo _logSegNo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ state->started_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!state->started_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8350,7 +8425,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* the backup taken during recovery is not available for the special
* recovery case described above.
*/
- if (!backup_started_in_recovery)
+ if (!state->started_in_recovery)
RequestXLogSwitch(false);
do
@@ -8385,13 +8460,13 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
- if (backup_started_in_recovery)
+ if (state->started_in_recovery)
{
XLogRecPtr recptr;
@@ -8404,7 +8479,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,16 +8511,17 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->startxlogfile, state->starttli, _logSegNo,
+ wal_segment_size);
/*
* Construct tablespace_map file.
@@ -8525,32 +8601,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(state->tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
@@ -8558,13 +8616,6 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8647,39 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates backup
+ * label contents and history file (if required), resets sessionBackupState
+ * and so on. It can optionally wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8720,10 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true && in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8759,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8771,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,8 +8783,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8770,14 +8793,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8809,30 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->stopxlogfile, state->stoptli, _logSegNo,
+ wal_segment_size);
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* construct history file contents */
+ create_backup_content_str(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8833,6 +8848,9 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
CleanupBackupHistory();
}
+ /* construct backup_label contents */
+ create_backup_content_str(state, false);
+
/*
* If archiving is enabled, wait for all the required WAL files to be
* archived before returning. If archiving isn't enabled, the required WAL
@@ -8855,15 +8873,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!in_recovery && XLogArchivingActive()) ||
+ (in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8923,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..96e575acc5 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -38,11 +38,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +58,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +69,18 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+ backup_state = get_backup_state(backupidstr);
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ do_pg_backup_start(backup_state, fast, NULL);
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,9 +101,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +118,14 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ free_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 2bb831a3cd..339cdfb099 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -231,9 +231,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +247,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = get_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -316,14 +315,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ create_backup_content_str(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -374,7 +378,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ free_backup_state(backup_state);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ebd321ba7..dcd44feab8 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlog_internal.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,12 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern BackupState get_backup_state(const char *name);
+extern void free_backup_state(BackupState state);
+extern void create_backup_content_str(BackupState state, bool forhistoryfile);
+extern void do_pg_backup_start(BackupState state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index 44291b337b..462520342a 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid)
}
#endif
+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name;
+ XLogRecPtr startpoint; /* backup start WAL location */
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ StringInfo backup_label; /* backup_label file contents. */
+ StringInfo tablespace_map; /* tablespace_map file contents. */
+ StringInfo history_file; /* history file contents. */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupStateData;
+
+typedef BackupStateData *BackupState;
+
/*
* Exported to support xlog switching from checkpointer
*/
--
2.34.1
On Wed, Sep 14, 2022 at 02:24:12PM +0530, Bharath Rupireddy wrote:
I'm attaching the v4 patch that's rebased on to the latest HEAD.
Please consider this for review.
I have been looking at this patch.
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState backup_state;
You could use initialize the state here with a {0}. That's simpler.
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid)
}
#endif
+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
Why is that in xlog_internal.h? This header includes a lot of
declarations about the internals of WAL, but the backup state is not
that internal. I'd like to think that we should begin separating the
backup-related routines into their own file, as of a set of
xlogbackup.c/h in this case. That's a split I have been wondering
about for some time now. The internals of xlog.c for the start/stop
backups are tied to XLogCtlData which map such a switch more
complicated than it looks, so we can begin small and have the routines
to create, free and build the label file and the tablespace map in
this new file.
+ state->name = (char *) palloc0(len + 1);
+ memcpy(state->name, name, len);
Or just pstrdup()?
+BackupState
+get_backup_state(const char *name)
+{
I would name this one create_backup_state() instead.
+void
+create_backup_content_str(BackupState state, bool forhistoryfile)
+{
This could be a build_backup_content().
It seems to me that there is no point in having the list of
tablespaces in BackupStateData? This actually makes the code harder
to follow, see for example the changes with do_pg_backup_start(), we
the list of tablespace may or may be not passed down as a pointer of
BackupStateData while BackupStateData is itself the first argument of
this routine. These are independent from the label and backup history
file, as well.
We need to be careful about the file format (see read_backup_label()),
and create_backup_content_str() is careful about that which is good.
Core does not care about the format of the backup history file, though
some community tools may. I agree that what you are proposing here
makes the generation of these files easier to follow, but let's
document what forhistoryfile is here for, at least. Saving the
the backup label and history file strings in BackupState is a
confusing interface IMO. It would be more intuitive to have the
backup state in input, and provide the string generated in output
depending on what we want from the backup state.
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ in_recovery = RecoveryInProgress();
[...]
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true && in_recovery == false)
I would have kept the naming to backup_started_in_recovery here. What
you are doing is logically right by relying on started_in_recovery to
check if recovery was running when the backup started, but this just
creates useless noise in the refactoring.
Something unrelated to your patch that I am noticing while scanning
the area is that we have been always lazy in freeing the label file
data allocated in TopMemoryContext when using the SQL functions if the
backup is aborted. We are not talking about this much amount of
memory each time a backup is performed, but that could be a cause for
memory bloat in a server if the same session is used and backups keep
failing, as the data is freed only on a successful pg_backup_stop().
Base backups through the replication protocol don't care about that as
the code keeps around the same pointer for the whole duration of
perform_base_backup(). Trying to tie the cleanup of the label file
with the abort phase would be the cause of more complications with
do_pg_backup_stop(), and xlog.c has no need to know about that now.
Just a remark for later.
--
Michael
On Fri, Sep 16, 2022 at 12:01 PM Michael Paquier <michael@paquier.xyz> wrote:
On Wed, Sep 14, 2022 at 02:24:12PM +0530, Bharath Rupireddy wrote:
I'm attaching the v4 patch that's rebased on to the latest HEAD.
Please consider this for review.I have been looking at this patch.
Thanks for reviewing it.
- StringInfo labelfile; - StringInfo tblspc_map_file; backup_manifest_info manifest; + BackupState backup_state; You could use initialize the state here with a {0}. That's simpler.
BackupState is a pointer to BackupStateData, we can't initialize that
way. However, I got rid of BackupStateData and used BackupState for
the structure directly, whenever pointer to the structure is required,
I'm using BackupState * to be more clear.
--- a/src/include/access/xlog_internal.h +++ b/src/include/access/xlog_internal.h @@ -380,6 +380,31 @@ GetRmgr(RmgrId rmid) } #endif+/* Structure to hold backup state. */
+typedef struct BackupStateData
+{
Why is that in xlog_internal.h? This header includes a lot of
declarations about the internals of WAL, but the backup state is not
that internal. I'd like to think that we should begin separating the
backup-related routines into their own file, as of a set of
xlogbackup.c/h in this case. That's a split I have been wondering
about for some time now. The internals of xlog.c for the start/stop
backups are tied to XLogCtlData which map such a switch more
complicated than it looks, so we can begin small and have the routines
to create, free and build the label file and the tablespace map in
this new file.
Good idea. It makes a lot more sense to me, because xlog.c is already
a file of 9000 LOC. I've created xlogbackup.c/.h files and added the
new code there. Once this patch gets in, I can offer my hand to move
do_pg_backup_start() and do_pg_backup_stop() from xlog.c and if okay,
pg_backup_start() and pg_backup_stop() from xlogfuncs.c to
xlogbackup.c/.h. Then, we might have to create new get/set APIs for
XLogCtl fields that do_pg_backup_start() and do_pg_backup_stop()
access.
+ state->name = (char *) palloc0(len + 1); + memcpy(state->name, name, len); Or just pstrdup()?
Done.
+BackupState +get_backup_state(const char *name) +{ I would name this one create_backup_state() instead.+void +create_backup_content_str(BackupState state, bool forhistoryfile) +{ This could be a build_backup_content().
I came up with more meaningful names - allocate_backup_state(),
deallocate_backup_state(), build_backup_content().
It seems to me that there is no point in having the list of
tablespaces in BackupStateData? This actually makes the code harder
to follow, see for example the changes with do_pg_backup_start(), we
the list of tablespace may or may be not passed down as a pointer of
BackupStateData while BackupStateData is itself the first argument of
this routine. These are independent from the label and backup history
file, as well.
I haven't stored the list of tablespaces in BackupState, it's the
string that do_pg_backup_start() creates is stored in there for
carrying it till pg_backup_stop(). Adding the tablespace_map,
backup_label, history_file in BackupState makes it easy to carry them
across various backup related functions.
We need to be careful about the file format (see read_backup_label()),
and create_backup_content_str() is careful about that which is good.
Core does not care about the format of the backup history file, though
some community tools may.
Are you suggesting that we need something like check_history_file()
similar to what read_backup_label() does by parsing each line of the
label file and erroring out if not in the required format?
I agree that what you are proposing here
makes the generation of these files easier to follow, but let's
document what forhistoryfile is here for, at least. Saving the
the backup label and history file strings in BackupState is a
confusing interface IMO. It would be more intuitive to have the
backup state in input, and provide the string generated in output
depending on what we want from the backup state.
We need to carry tablespace_map contents from do_pg_backup_start()
till pg_backup_stop(), backup_label and history_file too are easy to
carry across. Hence it will be good to have all of them i.e.
tablespace_map, backup_label and history_file in the BackupState
structure. IMO, this way is good.
- backup_started_in_recovery = RecoveryInProgress(); + Assert(state != NULL); + + in_recovery = RecoveryInProgress(); [...] - if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery) + if (state->started_in_recovery == true && in_recovery == false)I would have kept the naming to backup_started_in_recovery here. What
you are doing is logically right by relying on started_in_recovery to
check if recovery was running when the backup started, but this just
creates useless noise in the refactoring.
PSA new patch.
Something unrelated to your patch that I am noticing while scanning
the area is that we have been always lazy in freeing the label file
data allocated in TopMemoryContext when using the SQL functions if the
backup is aborted. We are not talking about this much amount of
memory each time a backup is performed, but that could be a cause for
memory bloat in a server if the same session is used and backups keep
failing, as the data is freed only on a successful pg_backup_stop().
Base backups through the replication protocol don't care about that as
the code keeps around the same pointer for the whole duration of
perform_base_backup(). Trying to tie the cleanup of the label file
with the abort phase would be the cause of more complications with
do_pg_backup_stop(), and xlog.c has no need to know about that now.
Just a remark for later.
Yeah, I think that can be solved by passing in backup_state to
do_pg_abort_backup(). If okay, I can work on this too as 0002 patch in
this thread or we can discuss this separately to get more attention
after this refactoring patch gets in.
I'm attaching v5 patch with the above review comments addressed,
please review it further.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v5-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v5-0001-Refactor-backup-related-code.patchDownload
From e940effe593b18cdd3c9c33dbf3ebaeb74f0b537 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sat, 17 Sep 2022 07:13:58 +0000
Subject: [PATCH v5] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 228 +++++++++---------------
src/backend/access/transam/xlogbackup.c | 98 ++++++++++
src/backend/access/transam/xlogfuncs.c | 43 ++---
src/backend/backup/basebackup.c | 32 ++--
src/include/access/xlog.h | 9 +-
src/include/access/xlogbackup.h | 56 ++++++
7 files changed, 275 insertions(+), 192 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 81d339d57d..2f6717e100 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,38 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState *state, bool fast, List **tablespaces)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8288,7 +8280,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8385,9 +8377,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8396,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,16 +8428,17 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->startxlogfile, state->starttli, _logSegNo,
+ wal_segment_size);
/*
* Construct tablespace_map file.
@@ -8525,46 +8518,23 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(state->tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8566,39 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates backup
+ * label contents and history file (if required), resets sessionBackupState
+ * and so on. It can optionally wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8639,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8679,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8691,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,8 +8703,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8770,14 +8713,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8729,30 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(state->stopxlogfile, state->stoptli, _logSegNo,
+ wal_segment_size);
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* construct history file contents */
+ build_backup_content(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8833,6 +8768,9 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
CleanupBackupHistory();
}
+ /* construct backup_label contents */
+ build_backup_content(state, false);
+
/*
* If archiving is enabled, wait for all the required WAL files to be
* archived before returning. If archiving isn't enabled, the required WAL
@@ -8855,15 +8793,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8843,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..3b01a4008b
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,98 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include <access/xlogbackup.h>
+
+/*
+ * Allocate backup state.
+ */
+BackupState *
+allocate_backup_state(const char *name)
+{
+ BackupState *state;
+
+ state = (BackupState *) palloc0(sizeof(BackupState));
+ state->name = pstrdup(name);
+ state->backup_label = makeStringInfo();
+ state->tablespace_map = makeStringInfo();
+ state->history_file = makeStringInfo();
+
+ return state;
+}
+
+/*
+ * Deallocate backup state.
+ */
+void
+deallocate_backup_state(BackupState *state)
+{
+ Assert(state != NULL);
+
+ pfree(state->name);
+ pfree(state->backup_label->data);
+ pfree(state->tablespace_map->data);
+ pfree(state->history_file->data);
+ pfree(state);
+}
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+
+ if (ishistoryfile)
+ str = state->history_file;
+ else
+ str = state->backup_label;
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->startxlogfile);
+
+ if (ishistoryfile)
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint),
+ state->stopxlogfile);
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..a7b69f7f1b 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -38,11 +39,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +59,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +70,18 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+ backup_state = allocate_backup_state(backupidstr);
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ do_pg_backup_start(backup_state, fast, NULL);
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,9 +102,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +119,14 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 2bb831a3cd..bc27279359 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +248,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = allocate_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -316,14 +316,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ build_backup_content(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -374,7 +379,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ deallocate_backup_state(backup_state);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ebd321ba7..6ca974615e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include <access/xlogbackup.h>
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,9 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(BackupState *state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..81c9dc2eab
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,56 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/*
+ * This macro is same as xlog_internals.h's MAXFNAMELEN. It may not be sensible
+ * to include xlog_internals.h just for this macro, hence defined a new one
+ * here.
+ *
+ * XXX: We can remove this macro, if at all, xlog_internals.h is included here
+ * in future.
+ */
+#define MAXFNAMELEN_BACKUP 64
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name; /* backup label name */
+ XLogRecPtr startpoint; /* backup start WAL location */
+ char startxlogfile[MAXFNAMELEN_BACKUP]; /* backup start WAL file */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ char stopxlogfile[MAXFNAMELEN_BACKUP]; /* backup stop WAL file */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+
+ StringInfo backup_label; /* backup_label file contents */
+ StringInfo tablespace_map; /* tablespace_map file contents */
+ StringInfo history_file; /* history file contents */
+} BackupState;
+
+extern BackupState *allocate_backup_state(const char *name);
+extern void deallocate_backup_state(BackupState *state);
+extern void build_backup_content(BackupState *state, bool ishistoryfile);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
On 2022/09/17 16:18, Bharath Rupireddy wrote:
Good idea. It makes a lot more sense to me, because xlog.c is already
a file of 9000 LOC. I've created xlogbackup.c/.h files and added the
new code there. Once this patch gets in, I can offer my hand to move
do_pg_backup_start() and do_pg_backup_stop() from xlog.c and if okay,
pg_backup_start() and pg_backup_stop() from xlogfuncs.c to
xlogbackup.c/.h. Then, we might have to create new get/set APIs for
XLogCtl fields that do_pg_backup_start() and do_pg_backup_stop()
access.
The definition of SessionBackupState enum type also should be in xlogbackup.h?
We need to carry tablespace_map contents from do_pg_backup_start()
till pg_backup_stop(), backup_label and history_file too are easy to
carry across. Hence it will be good to have all of them i.e.
tablespace_map, backup_label and history_file in the BackupState
structure. IMO, this way is good.
backup_label and history_file are not carried between pg_backup_start()
and _stop(), so don't need to be saved in BackupState. Their contents
can be easily created from other saved fields in BackupState,
if necessary. So at least for me it's better to get rid of them from
BackupState and don't allocate TopMemoryContext memory for them.
Something unrelated to your patch that I am noticing while scanning
the area is that we have been always lazy in freeing the label file
data allocated in TopMemoryContext when using the SQL functions if the
backup is aborted. We are not talking about this much amount of
memory each time a backup is performed, but that could be a cause for
memory bloat in a server if the same session is used and backups keep
failing, as the data is freed only on a successful pg_backup_stop().
Base backups through the replication protocol don't care about that as
the code keeps around the same pointer for the whole duration of
perform_base_backup(). Trying to tie the cleanup of the label file
with the abort phase would be the cause of more complications with
do_pg_backup_stop(), and xlog.c has no need to know about that now.
Just a remark for later.Yeah, I think that can be solved by passing in backup_state to
do_pg_abort_backup(). If okay, I can work on this too as 0002 patch in
this thread or we can discuss this separately to get more attention
after this refactoring patch gets in.
Or, to avoid such memory bloat, how about allocating the memory for
backup_state only when it's NULL?
I'm attaching v5 patch with the above review comments addressed,
please review it further.
Thanks for updating the patch!
+ char startxlogfile[MAXFNAMELEN_BACKUP]; /* backup start WAL file */
<snip>
+ char stopxlogfile[MAXFNAMELEN_BACKUP]; /* backup stop WAL file */
These file names seem not necessary in BackupState because they can be
calculated from other fields like startpoint and starttli, etc when
making backup_label and history file contents. If we remove them from
BackupState, we can also remove the definition of MAXFNAMELEN_BACKUP
macro from xlogbackup.h.
+ /* construct backup_label contents */
+ build_backup_content(state, false);
In basebackup case, build_backup_content() is called unnecessary twice
because do_pg_stop_backup() and its caller, perform_base_backup() call
that. This makes me think that it's better to get rid of the call to
build_backup_content() from do_pg_backup_stop(). Instead its callers,
perform_base_backup() and pg_backup_stop() should call that.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On Sun, Sep 18, 2022 at 7:38 AM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
On 2022/09/17 16:18, Bharath Rupireddy wrote:
Good idea. It makes a lot more sense to me, because xlog.c is already
a file of 9000 LOC. I've created xlogbackup.c/.h files and added the
new code there. Once this patch gets in, I can offer my hand to move
do_pg_backup_start() and do_pg_backup_stop() from xlog.c and if okay,
pg_backup_start() and pg_backup_stop() from xlogfuncs.c to
xlogbackup.c/.h. Then, we might have to create new get/set APIs for
XLogCtl fields that do_pg_backup_start() and do_pg_backup_stop()
access.The definition of SessionBackupState enum type also should be in xlogbackup.h?
Correct. Basically, all the backup related code from xlog.c,
xlogfuncs.c and elsewhere can go to xlogbackup.c/.h. I will focus on
that refactoring patch once this gets in.
We need to carry tablespace_map contents from do_pg_backup_start()
till pg_backup_stop(), backup_label and history_file too are easy to
carry across. Hence it will be good to have all of them i.e.
tablespace_map, backup_label and history_file in the BackupState
structure. IMO, this way is good.backup_label and history_file are not carried between pg_backup_start()
and _stop(), so don't need to be saved in BackupState. Their contents
can be easily created from other saved fields in BackupState,
if necessary. So at least for me it's better to get rid of them from
BackupState and don't allocate TopMemoryContext memory for them.
Yeah, but they have to be carried from do_pg_backup_stop() to
pg_backup_stop() or callers and also instead of keeping tablespace_map
in BackupState and others elsewhere don't seem to be a good idea to
me. IMO, BackupState is a good place to contain all the information
that's carried across various functions. I've changed the code to
lazily (upon first use in the backend) allocate memory for all of them
as we're concerned of the memory allocation beforehand.
Yeah, I think that can be solved by passing in backup_state to
do_pg_abort_backup(). If okay, I can work on this too as 0002 patch in
this thread or we can discuss this separately to get more attention
after this refactoring patch gets in.Or, to avoid such memory bloat, how about allocating the memory for
backup_state only when it's NULL?
Ah my bad, I missed that. Done now.
I'm attaching v5 patch with the above review comments addressed,
please review it further.Thanks for updating the patch!
Thanks for reviewing it.
+ char startxlogfile[MAXFNAMELEN_BACKUP]; /* backup start WAL file */ <snip> + char stopxlogfile[MAXFNAMELEN_BACKUP]; /* backup stop WAL file */These file names seem not necessary in BackupState because they can be
calculated from other fields like startpoint and starttli, etc when
making backup_label and history file contents. If we remove them from
BackupState, we can also remove the definition of MAXFNAMELEN_BACKUP
macro from xlogbackup.h.
Done.
+ /* construct backup_label contents */ + build_backup_content(state, false);In basebackup case, build_backup_content() is called unnecessary twice
because do_pg_stop_backup() and its caller, perform_base_backup() call
that. This makes me think that it's better to get rid of the call to
build_backup_content() from do_pg_backup_stop(). Instead its callers,
perform_base_backup() and pg_backup_stop() should call that.
Yeah, it's a good idea. Done that way. It's easier because we can
create backup_label file contents at any point of time after
pg_backup_start().
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v6-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v6-0001-Refactor-backup-related-code.patchDownload
From 48e45e8d4f4795d9691b37e6a70485059774bba0 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sun, 18 Sep 2022 07:50:59 +0000
Subject: [PATCH v6] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 229 ++++++++----------------
src/backend/access/transam/xlogbackup.c | 183 +++++++++++++++++++
src/backend/access/transam/xlogfuncs.c | 58 +++---
src/backend/backup/basebackup.c | 33 ++--
src/include/access/xlog.h | 9 +-
src/include/access/xlogbackup.h | 49 +++++
7 files changed, 362 insertions(+), 200 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 81d339d57d..239c05f9b9 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,37 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState *state, bool fast, List **tablespaces)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8288,7 +8279,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8385,9 +8376,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8395,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,17 +8427,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
* Construct tablespace_map file.
*/
@@ -8525,46 +8513,22 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
- ti->oid, escapedpath.data);
-
+ build_backup_tablespace_map_content(state, ti->oid,
+ escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8560,42 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
+ *
+ * It is the responsibility of the caller to create backup label contents by
+ * calling build_backup_content().
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8636,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8676,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8688,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,8 +8700,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8770,14 +8710,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8726,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Construct history file contents */
+ build_backup_content(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8855,15 +8784,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8834,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..a71942ab31
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,183 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
+#include "utils/memutils.h"
+
+/*
+ * Allocate backup state.
+ *
+ * Note that the caller has to ensure right memory context is set for the
+ * backup state.
+ */
+BackupState *
+allocate_backup_state(const char *name)
+{
+ BackupState *state;
+
+ state = (BackupState *) palloc0(sizeof(BackupState));
+ state->name = pstrdup(name);
+
+ return state;
+}
+
+/*
+ * Thin wrapper for allocating memory for StringInfo members of backup state
+ * structure.
+ *
+ * This function allocates the member in the same memory context in which
+ * backup state was allocated.
+ */
+StringInfo
+allocate_backup_state_member(BackupState *state, StringInfo member)
+{
+ MemoryContext oldcontext;
+
+ Assert(state != NULL);
+
+ if (member != NULL)
+ pfree(member);
+
+ oldcontext = MemoryContextSwitchTo(GetMemoryChunkContext(state));
+ member = makeStringInfo();
+ MemoryContextSwitchTo(oldcontext);
+
+ return member;
+}
+
+/*
+ * Deallocate backup state.
+ */
+void
+deallocate_backup_state(BackupState *state)
+{
+ if (state == NULL)
+ return; /* nothing to do */
+
+ if (state->name != NULL)
+ {
+ pfree(state->name);
+ state->name = NULL;
+ }
+
+ if (state->backup_label != NULL)
+ {
+ pfree(state->backup_label->data);
+ state->backup_label = NULL;
+ }
+
+ if (state->tablespace_map != NULL)
+ {
+ pfree(state->tablespace_map->data);
+ state->tablespace_map = NULL;
+ }
+
+ if (state->history_file != NULL)
+ {
+ pfree(state->history_file->data);
+ state->history_file = NULL;
+ }
+
+ pfree(state);
+}
+
+/*
+ * Construct tablespace_map file contents
+ */
+void
+build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath)
+{
+ Assert(state != NULL);
+
+ state->tablespace_map =
+ allocate_backup_state_member(state, state->tablespace_map);
+
+ appendStringInfo(state->tablespace_map, "%s %s\n", oid, escapedpath);
+}
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+
+ Assert(state != NULL);
+
+ if (ishistoryfile)
+ {
+ state->history_file =
+ allocate_backup_state_member(state, state->history_file);
+
+ str = state->history_file;
+ }
+ else
+ {
+ state->backup_label =
+ allocate_backup_state_member(state, state->backup_label);
+
+ str = state->backup_label;
+ }
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..c4dd39e8f4 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -38,11 +39,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +59,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +70,24 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
- oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
- MemoryContextSwitchTo(oldcontext);
+ if (backup_state == NULL)
+ {
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ backup_state = allocate_backup_state(backupidstr);
+ MemoryContextSwitchTo(oldcontext);
+ }
+ else
+ {
+ MemSet(backup_state, 0, sizeof(BackupState));
+ }
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backup_state, fast, NULL);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,9 +108,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +125,17 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /* Construct backup_label contents */
+ build_backup_content(backup_state, false);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 1f1cff1a58..b38a739ab1 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +248,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = allocate_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -316,14 +316,19 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ build_backup_content(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -374,7 +379,13 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ebd321ba7..6ca974615e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include <access/xlogbackup.h>
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,9 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(BackupState *state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..bfdeca6922
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name; /* backup label name */
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+
+ StringInfo backup_label; /* backup_label file contents */
+ StringInfo tablespace_map; /* tablespace_map file contents */
+ StringInfo history_file; /* history file contents */
+} BackupState;
+
+extern BackupState *allocate_backup_state(const char *name);
+extern StringInfo allocate_backup_state_member(BackupState *state,
+ StringInfo member);
+extern void deallocate_backup_state(BackupState *state);
+extern void build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath);
+extern void build_backup_content(BackupState *state, bool ishistoryfile);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
On Sun, Sep 18, 2022 at 1:23 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
cfbot fails [1]https://cirrus-ci.com/task/5816966114967552?logs=test_world#L720 with v6 patch. I made a silly mistake by not checking
the output of "make check-world -j 16" fully, I just saw the end
message "All tests successful." before posting the v6 patch.
The failure is due to perform_base_backup() accessing BackupState's
tablespace_map without a null check, so I fixed it.
Sorry for the noise. Please review the attached v7 patch further.
[1]: https://cirrus-ci.com/task/5816966114967552?logs=test_world#L720
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v7-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v7-0001-Refactor-backup-related-code.patchDownload
From 3195456d869a58550a41df2ff0e3f6e3d90a3dcd Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Sun, 18 Sep 2022 12:54:12 +0000
Subject: [PATCH v7] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 229 ++++++++----------------
src/backend/access/transam/xlogbackup.c | 183 +++++++++++++++++++
src/backend/access/transam/xlogfuncs.c | 58 +++---
src/backend/backup/basebackup.c | 36 ++--
src/include/access/xlog.h | 9 +-
src/include/access/xlogbackup.h | 49 +++++
7 files changed, 364 insertions(+), 201 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 81d339d57d..239c05f9b9 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,37 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState *state, bool fast, List **tablespaces)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8288,7 +8279,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8385,9 +8376,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8395,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,17 +8427,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
* Construct tablespace_map file.
*/
@@ -8525,46 +8513,22 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
- ti->oid, escapedpath.data);
-
+ build_backup_tablespace_map_content(state, ti->oid,
+ escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8560,42 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
+ *
+ * It is the responsibility of the caller to create backup label contents by
+ * calling build_backup_content().
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8636,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8676,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8688,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,8 +8700,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8770,14 +8710,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8726,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Construct history file contents */
+ build_backup_content(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8855,15 +8784,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8834,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..a71942ab31
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,183 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
+#include "utils/memutils.h"
+
+/*
+ * Allocate backup state.
+ *
+ * Note that the caller has to ensure right memory context is set for the
+ * backup state.
+ */
+BackupState *
+allocate_backup_state(const char *name)
+{
+ BackupState *state;
+
+ state = (BackupState *) palloc0(sizeof(BackupState));
+ state->name = pstrdup(name);
+
+ return state;
+}
+
+/*
+ * Thin wrapper for allocating memory for StringInfo members of backup state
+ * structure.
+ *
+ * This function allocates the member in the same memory context in which
+ * backup state was allocated.
+ */
+StringInfo
+allocate_backup_state_member(BackupState *state, StringInfo member)
+{
+ MemoryContext oldcontext;
+
+ Assert(state != NULL);
+
+ if (member != NULL)
+ pfree(member);
+
+ oldcontext = MemoryContextSwitchTo(GetMemoryChunkContext(state));
+ member = makeStringInfo();
+ MemoryContextSwitchTo(oldcontext);
+
+ return member;
+}
+
+/*
+ * Deallocate backup state.
+ */
+void
+deallocate_backup_state(BackupState *state)
+{
+ if (state == NULL)
+ return; /* nothing to do */
+
+ if (state->name != NULL)
+ {
+ pfree(state->name);
+ state->name = NULL;
+ }
+
+ if (state->backup_label != NULL)
+ {
+ pfree(state->backup_label->data);
+ state->backup_label = NULL;
+ }
+
+ if (state->tablespace_map != NULL)
+ {
+ pfree(state->tablespace_map->data);
+ state->tablespace_map = NULL;
+ }
+
+ if (state->history_file != NULL)
+ {
+ pfree(state->history_file->data);
+ state->history_file = NULL;
+ }
+
+ pfree(state);
+}
+
+/*
+ * Construct tablespace_map file contents
+ */
+void
+build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath)
+{
+ Assert(state != NULL);
+
+ state->tablespace_map =
+ allocate_backup_state_member(state, state->tablespace_map);
+
+ appendStringInfo(state->tablespace_map, "%s %s\n", oid, escapedpath);
+}
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char stopstrfbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+
+ Assert(state != NULL);
+
+ if (ishistoryfile)
+ {
+ state->history_file =
+ allocate_backup_state_member(state, state->history_file);
+
+ str = state->history_file;
+ }
+ else
+ {
+ state->backup_label =
+ allocate_backup_state_member(state, state->backup_label);
+
+ str = state->backup_label;
+ }
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..c4dd39e8f4 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -38,11 +39,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,7 +59,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +70,24 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
- oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
- MemoryContextSwitchTo(oldcontext);
+ if (backup_state == NULL)
+ {
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ backup_state = allocate_backup_state(backupidstr);
+ MemoryContextSwitchTo(oldcontext);
+ }
+ else
+ {
+ MemSet(backup_state, 0, sizeof(BackupState));
+ }
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backup_state, fast, NULL);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,9 +108,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -127,19 +125,17 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /* Construct backup_label contents */
+ build_backup_content(backup_state, false);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 1f1cff1a58..81cf39b5f2 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +248,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = allocate_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -316,14 +316,20 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ build_backup_content(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
- if (opt->sendtblspcmapfile)
+ if (opt->sendtblspcmapfile &&
+ backup_state->tablespace_map != NULL)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -374,7 +380,13 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ebd321ba7..6ca974615e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include <access/xlogbackup.h>
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,9 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(BackupState *state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..bfdeca6922
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+ char *name; /* backup label name */
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+
+ StringInfo backup_label; /* backup_label file contents */
+ StringInfo tablespace_map; /* tablespace_map file contents */
+ StringInfo history_file; /* history file contents */
+} BackupState;
+
+extern BackupState *allocate_backup_state(const char *name);
+extern StringInfo allocate_backup_state_member(BackupState *state,
+ StringInfo member);
+extern void deallocate_backup_state(BackupState *state);
+extern void build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath);
+extern void build_backup_content(BackupState *state, bool ishistoryfile);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
On 2022/09/18 23:09, Bharath Rupireddy wrote:
On Sun, Sep 18, 2022 at 1:23 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:cfbot fails [1] with v6 patch. I made a silly mistake by not checking
the output of "make check-world -j 16" fully, I just saw the end
message "All tests successful." before posting the v6 patch.The failure is due to perform_base_backup() accessing BackupState's
tablespace_map without a null check, so I fixed it.Sorry for the noise. Please review the attached v7 patch further.
Thanks for updating the patch!
=# SELECT * FROM pg_backup_start('test', true);
=# SELECT * FROM pg_backup_stop();
LOG: server process (PID 15651) was terminated by signal 11: Segmentation fault: 11
DETAIL: Failed process was running: SELECT * FROM pg_backup_stop();
With v7 patch, pg_backup_stop() caused the segmentation fault.
=# SELECT * FROM pg_backup_start(repeat('test', 1024));
ERROR: backup label too long (max 1024 bytes)
STATEMENT: SELECT * FROM pg_backup_start(repeat('test', 1024));
=# SELECT * FROM pg_backup_start(repeat('testtest', 1024));
LOG: server process (PID 15844) was terminated by signal 11: Segmentation fault: 11
DETAIL: Failed process was running: SELECT * FROM pg_backup_start(repeat('testtest', 1024));
When I specified longer backup label in the second run of pg_backup_start()
after the first run failed, it caused the segmentation fault.
+ state = (BackupState *) palloc0(sizeof(BackupState));
+ state->name = pstrdup(name);
pg_backup_start() calls allocate_backup_state() and allocates the memory for
the input backup label before checking its length in do_pg_backup_start().
This can cause the memory for backup label to be allocated too much
unnecessary. I think that the maximum length of BackupState.name should
be MAXPGPATH (i.e., maximum allowed length for backup label).
The definition of SessionBackupState enum type also should be in xlogbackup.h?
Correct. Basically, all the backup related code from xlog.c,
xlogfuncs.c and elsewhere can go to xlogbackup.c/.h. I will focus on
that refactoring patch once this gets in.
Understood.
Yeah, but they have to be carried from do_pg_backup_stop() to
pg_backup_stop() or callers and also instead of keeping tablespace_map
in BackupState and others elsewhere don't seem to be a good idea to
me. IMO, BackupState is a good place to contain all the information
that's carried across various functions.
In v7 patch, since pg_backup_stop() calls build_backup_content(),
backup_label and history_file seem not to be carried from do_pg_backup_stop()
to pg_backup_stop(). This makes me still think that it's better not to include
them in BackupState...
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On Mon, Sep 19, 2022 at 2:38 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
Thanks for updating the patch!
=# SELECT * FROM pg_backup_start('test', true);
=# SELECT * FROM pg_backup_stop();
LOG: server process (PID 15651) was terminated by signal 11: Segmentation fault: 11
DETAIL: Failed process was running: SELECT * FROM pg_backup_stop();With v7 patch, pg_backup_stop() caused the segmentation fault.
Fixed. I believed that the regression tests cover pg_backup_start()
and pg_backup_stop(), and relied on make check-world, surprisingly
there's no test that covers these functions. Is it a good idea to add
tests for these functions in misc_functions.sql or backup.sql or
somewhere so that they get run as part of make check? Thoughts?
=# SELECT * FROM pg_backup_start(repeat('test', 1024));
ERROR: backup label too long (max 1024 bytes)
STATEMENT: SELECT * FROM pg_backup_start(repeat('test', 1024));=# SELECT * FROM pg_backup_start(repeat('testtest', 1024));
LOG: server process (PID 15844) was terminated by signal 11: Segmentation fault: 11
DETAIL: Failed process was running: SELECT * FROM pg_backup_start(repeat('testtest', 1024));When I specified longer backup label in the second run of pg_backup_start()
after the first run failed, it caused the segmentation fault.+ state = (BackupState *) palloc0(sizeof(BackupState)); + state->name = pstrdup(name);pg_backup_start() calls allocate_backup_state() and allocates the memory for
the input backup label before checking its length in do_pg_backup_start().
This can cause the memory for backup label to be allocated too much
unnecessary. I think that the maximum length of BackupState.name should
be MAXPGPATH (i.e., maximum allowed length for backup label).
That's a good idea. I'm marking a flag if the label name overflows (>
MAXPGPATH), later using it in do_pg_backup_start() to error out. We
could've thrown error directly, but that changes the behaviour - right
now, first "
wal_level must be set to \"replica\" or \"logical\" at server start."
gets emitted and then label name overflow error - I don't want to do
that.
Yeah, but they have to be carried from do_pg_backup_stop() to
pg_backup_stop() or callers and also instead of keeping tablespace_map
in BackupState and others elsewhere don't seem to be a good idea to
me. IMO, BackupState is a good place to contain all the information
that's carried across various functions.In v7 patch, since pg_backup_stop() calls build_backup_content(),
backup_label and history_file seem not to be carried from do_pg_backup_stop()
to pg_backup_stop(). This makes me still think that it's better not to include
them in BackupState...
I'm a bit worried about the backup state being spread across if we
separate out backup_label and history_file from BackupState and keep
tablespace_map contents there. As I said upthread, we are not
allocating memory for them at the beginning, we allocate only when
needed. IMO, this code is readable and more extensible.
I've also taken help of the error callback mechanism to clean up the
allocated memory in case of a failure. For do_pg_abort_backup() cases,
I think we don't need to clean the memory because that gets called on
proc exit (before_shmem_exit()).
Please review the v8 patch further.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v8-0001-Refactor-backup-related-code.patchapplication/octet-stream; name=v8-0001-Refactor-backup-related-code.patchDownload
From d2f1cb87ec1182194ddd9618c4e0f3c78da4b12e Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Mon, 19 Sep 2022 12:17:38 +0000
Subject: [PATCH v8] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 229 +++++++--------------
src/backend/access/transam/xlogbackup.c | 253 ++++++++++++++++++++++++
src/backend/access/transam/xlogfuncs.c | 84 +++++---
src/backend/backup/basebackup.c | 36 ++--
src/include/access/xlog.h | 9 +-
src/include/access/xlogbackup.h | 54 +++++
7 files changed, 466 insertions(+), 200 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 81d339d57d..0a89d4ad8b 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,37 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster) and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.)
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
+ * The tablespace map contents are appended to backup state's tablespace_map
+ * and the caller is responsible for including it in the backup archive as
+ * 'tablespace_map'. The tablespace_map file is required mainly for tar format
+ * in windows as native windows utilities are not able to create symlinks while
+ * extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
* Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup, and the corresponding timeline ID via backup state startpoint and
+ * starttli respectively.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState *state, bool fast, List **tablespaces)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8288,7 +8279,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (state->name_overflowed == true)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8385,9 +8376,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8395,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,17 +8427,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
* Construct tablespace_map file.
*/
@@ -8525,46 +8513,22 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
- ti->oid, escapedpath.data);
-
+ build_backup_tablespace_map_content(state, ti->oid,
+ escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8560,42 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
* Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup, and the corresponding timeline ID via backup state stoppoint and
+ * stoptli respectively.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
+ *
+ * It is the responsibility of the caller to create backup label contents by
+ * calling build_backup_content().
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8636,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8676,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8688,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,8 +8700,8 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
@@ -8770,14 +8710,15 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8726,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Construct history file contents */
+ build_backup_content(state, true);
+
+ fprintf(fp, "%s", state->history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8855,15 +8784,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8834,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..35094f2f59
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,253 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
+#include "utils/memutils.h"
+
+static void deallocate_backup_state_members(BackupState *state);
+
+/*
+ * Allocate backup state.
+ *
+ * Note that the caller has to ensure right memory context is set for the
+ * backup state.
+ */
+BackupState *
+allocate_backup_state(const char *name)
+{
+ BackupState *state;
+ Size len;
+
+ state = (BackupState *) palloc0(sizeof(BackupState));
+
+ len = strlen(name);
+
+ /*
+ * If user entered a backup label name of length more than MAXPGPATH bytes,
+ * just set a flag to later error out in do_pg_backup_start().
+ */
+ if (len > MAXPGPATH)
+ {
+ state->name_overflowed = true;
+ MemSet(state->name, '\0', sizeof(state->name));
+ }
+ else
+ memcpy(state->name, name, len);
+
+ return state;
+}
+
+/*
+ * Reset backup state members.
+ */
+void
+reset_backup_state(BackupState *state, const char *name)
+{
+ Size len;
+
+ Assert(state != NULL);
+
+ deallocate_backup_state_members(state);
+
+ MemSet(state, 0, sizeof(BackupState));
+ MemSet(state->name, '\0', sizeof(state->name));
+
+ len = strlen(name);
+
+ /*
+ * If user entered a backup label name of length more than MAXPGPATH bytes,
+ * just set a flag to later error out in do_pg_backup_start().
+ */
+ if (len > MAXPGPATH)
+ state->name_overflowed = true;
+ else
+ memcpy(state->name, name, len);
+}
+
+/*
+ * Thin wrapper for allocating memory for StringInfo members of backup state
+ * structure.
+ *
+ * This function allocates the member in the same memory context in which
+ * backup state was allocated.
+ */
+StringInfo
+allocate_backup_state_member(BackupState *state, StringInfo member)
+{
+ MemoryContext oldcontext;
+
+ Assert(state != NULL);
+
+ if (member != NULL)
+ pfree(member);
+
+ oldcontext = MemoryContextSwitchTo(GetMemoryChunkContext(state));
+ member = makeStringInfo();
+ MemoryContextSwitchTo(oldcontext);
+
+ return member;
+}
+
+/*
+ * Deallocate backup state members.
+ */
+static void
+deallocate_backup_state_members(BackupState *state)
+{
+ Assert(state != NULL);
+
+ if (state->backup_label != NULL)
+ {
+ pfree(state->backup_label->data);
+ state->backup_label = NULL;
+ }
+
+ if (state->tablespace_map != NULL)
+ {
+ pfree(state->tablespace_map->data);
+ state->tablespace_map = NULL;
+ }
+
+ if (state->history_file != NULL)
+ {
+ pfree(state->history_file->data);
+ state->history_file = NULL;
+ }
+}
+
+/*
+ * Deallocate backup state along with its members.
+ */
+void
+deallocate_backup_state(BackupState *state)
+{
+ if (state == NULL)
+ return; /* nothing to do */
+
+ deallocate_backup_state_members(state);
+
+ pfree(state);
+}
+
+/*
+ * Error context callback for errors occurring during pg_backup_start() to free
+ * up memory allocated for backup sate members.
+ */
+void
+backup_state_members_error_callback(void *arg)
+{
+ BackupState *state = (BackupState *) arg;
+
+ if (state == NULL)
+ return; /* nothing to do */
+
+ /*
+ * Note that we don't deallocate backup state, if we do, caller will have a
+ * dangling pointer. Hence we just deallocate backup state members.
+ */
+ deallocate_backup_state_members(state);
+}
+
+/*
+ * Construct tablespace_map file contents
+ */
+void
+build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath)
+{
+ Assert(state != NULL);
+
+ state->tablespace_map =
+ allocate_backup_state_member(state, state->tablespace_map);
+
+ appendStringInfo(state->tablespace_map, "%s %s\n", oid, escapedpath);
+}
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ StringInfo str;
+ char startstrbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+
+ Assert(state != NULL);
+
+ if (ishistoryfile)
+ {
+ state->history_file =
+ allocate_backup_state_member(state, state->history_file);
+
+ str = state->history_file;
+ }
+ else
+ {
+ state->backup_label =
+ allocate_backup_state_member(state, state->backup_label);
+
+ str = state->backup_label;
+ }
+
+ Assert(str != NULL);
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ char stopstrfbuf[128];
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..80f984cb31 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -38,11 +39,7 @@
#include "utils/timestamp.h"
#include "utils/tuplestore.h"
-/*
- * Store label file and tablespace map during backups.
- */
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
@@ -62,9 +59,9 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
+ ErrorContextCallback errcallback;
backupidstr = text_to_cstring(backupid);
@@ -74,20 +71,33 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state need to be long-lived as its contents are read in
+ * pg_backup_stop, hence allocate in TopMemoryContext.
*/
- oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
- MemoryContextSwitchTo(oldcontext);
+ if (backup_state == NULL)
+ {
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+ backup_state = allocate_backup_state(backupidstr);
+ MemoryContextSwitchTo(oldcontext);
+ }
+ else
+ {
+ reset_backup_state(backup_state, backupidstr);
+ }
+
+ /* Set up callback to clean up backup_state members */
+ errcallback.callback = backup_state_members_error_callback;
+ errcallback.arg = (void *) backup_state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backup_state, fast, NULL);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -108,10 +118,15 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
SessionBackupState status = get_backup_status();
+ ErrorContextCallback errcallback;
+
+ /* Set up callback to clean up backup_state members */
+ errcallback.callback = backup_state_members_error_callback;
+ errcallback.arg = (void *) backup_state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
/* Initialize attributes information in the tuple descriptor */
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -127,19 +142,28 @@ pg_backup_stop(PG_FUNCTION_ARGS)
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
+ /* Construct backup_label contents */
+ build_backup_content(backup_state, false);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+
+ if (backup_state->backup_label != NULL)
+ values[1] = CStringGetTextDatum(backup_state->backup_label->data);
+ else
+ nulls[1] = true;
+
+ if (backup_state->tablespace_map != NULL)
+ values[2] = CStringGetTextDatum(backup_state->tablespace_map->data);
+ else
+ nulls[2] = true;
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 1f1cff1a58..81cf39b5f2 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +248,18 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ backup_state = allocate_backup_state(opt->label);
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -316,14 +316,20 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
+ /* construct backup_label contents */
+ build_backup_content(backup_state, false);
+
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_state->backup_label->data,
&manifest);
/* Then the tablespace_map file, if required... */
- if (opt->sendtblspcmapfile)
+ if (opt->sendtblspcmapfile &&
+ backup_state->tablespace_map != NULL)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ backup_state->tablespace_map->data,
&manifest);
sendtblspclinks = false;
}
@@ -374,7 +380,13 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ deallocate_backup_state(backup_state);
+ backup_state = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ebd321ba7..6ca974615e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include <access/xlogbackup.h>
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,9 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(BackupState *state, bool fast,
+ List **tablespaces);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..a19643e8ef
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,54 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+
+ /* Null-terminated backup label name. */
+ char name[MAXPGPATH + 1];
+ bool name_overflowed; /* is backup label name overflowed? */
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+
+ StringInfo backup_label; /* backup_label file contents */
+ StringInfo tablespace_map; /* tablespace_map file contents */
+ StringInfo history_file; /* history file contents */
+} BackupState;
+
+extern BackupState *allocate_backup_state(const char *name);
+extern void reset_backup_state(BackupState *state, const char *name);
+extern StringInfo allocate_backup_state_member(BackupState *state,
+ StringInfo member);
+extern void deallocate_backup_state(BackupState *state);
+extern void build_backup_tablespace_map_content(BackupState *state,
+ char *oid,
+ char *escapedpath);
+extern void build_backup_content(BackupState *state, bool ishistoryfile);
+extern void backup_state_members_error_callback(void *arg);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
On Mon, Sep 19, 2022 at 06:26:34PM +0530, Bharath Rupireddy wrote:
On Mon, Sep 19, 2022 at 2:38 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
Fixed. I believed that the regression tests cover pg_backup_start()
and pg_backup_stop(), and relied on make check-world, surprisingly
there's no test that covers these functions. Is it a good idea to add
tests for these functions in misc_functions.sql or backup.sql or
somewhere so that they get run as part of make check? Thoughts?
The main regression test suite should not include direct calls to
pg_backup_start() or pg_backup_stop() as these depend on wal_level,
and we spend a certain amount of resources in keeping the tests a
maximum portable across such configurations, wal_level = minimal being
one of them. One portable example is in 001_stream_rep.pl.
That's a good idea. I'm marking a flag if the label name overflows (>
MAXPGPATH), later using it in do_pg_backup_start() to error out. We
could've thrown error directly, but that changes the behaviour - right
now, first "
wal_level must be set to \"replica\" or \"logical\" at server start."
gets emitted and then label name overflow error - I don't want to do
that.
- if (strlen(backupidstr) > MAXPGPATH)
+ if (state->name_overflowed == true)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
It does not strike me as a huge issue to force a truncation of such
backup label names. 1024 is large enough for basically all users,
in my opinion. Or you could just truncate in the internal logic, but
still complain at MAXPGPATH - 1 as the last byte would be for the zero
termination. In short, there is no need to complicate things with
name_overflowed.
In v7 patch, since pg_backup_stop() calls build_backup_content(),
backup_label and history_file seem not to be carried from do_pg_backup_stop()
to pg_backup_stop(). This makes me still think that it's better not to include
them in BackupState...I'm a bit worried about the backup state being spread across if we
separate out backup_label and history_file from BackupState and keep
tablespace_map contents there. As I said upthread, we are not
allocating memory for them at the beginning, we allocate only when
needed. IMO, this code is readable and more extensible.
+ StringInfo backup_label; /* backup_label file contents */
+ StringInfo tablespace_map; /* tablespace_map file contents */
+ StringInfo history_file; /* history file contents */
IMV, repeating a point I already made once upthread, BackupState
should hold none of these. Let's just generate the contents of these
files in the contexts where they are needed, making BackupState
something to rely on to build them in the code paths where they are
necessary. This is going to make the reasoning around the memory
contexts where each one of them is stored much easier and reduce the
changes of bugs in the long-term.
I've also taken help of the error callback mechanism to clean up the
allocated memory in case of a failure. For do_pg_abort_backup() cases,
I think we don't need to clean the memory because that gets called on
proc exit (before_shmem_exit()).
Memory could still bloat while the process running the SQL functions
is running depending on the error code path, anyway.
--
Michael
On Tue, Sep 20, 2022 at 7:20 AM Michael Paquier <michael@paquier.xyz> wrote:
The main regression test suite should not include direct calls to
pg_backup_start() or pg_backup_stop() as these depend on wal_level,
and we spend a certain amount of resources in keeping the tests a
maximum portable across such configurations, wal_level = minimal being
one of them. One portable example is in 001_stream_rep.pl.
Understood.
That's a good idea. I'm marking a flag if the label name overflows (>
MAXPGPATH), later using it in do_pg_backup_start() to error out. We
could've thrown error directly, but that changes the behaviour - right
now, first "
wal_level must be set to \"replica\" or \"logical\" at server start."
gets emitted and then label name overflow error - I don't want to do
that.- if (strlen(backupidstr) > MAXPGPATH)
+ if (state->name_overflowed == true)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
It does not strike me as a huge issue to force a truncation of such
backup label names. 1024 is large enough for basically all users,
in my opinion. Or you could just truncate in the internal logic, but
still complain at MAXPGPATH - 1 as the last byte would be for the zero
termination. In short, there is no need to complicate things with
name_overflowed.
We currently allow MAXPGPATH bytes of label name excluding null
termination. I don't want to change this behaviour. In the attached v9
patch, I'm truncating the larger names to MAXPGPATH + 1 bytes in
backup state (one extra byte for representing that the name has
overflown, and another extra byte for null termination).
+ StringInfo backup_label; /* backup_label file contents */ + StringInfo tablespace_map; /* tablespace_map file contents */ + StringInfo history_file; /* history file contents */ IMV, repeating a point I already made once upthread, BackupState should hold none of these. Let's just generate the contents of these files in the contexts where they are needed, making BackupState something to rely on to build them in the code paths where they are necessary. This is going to make the reasoning around the memory contexts where each one of them is stored much easier and reduce the changes of bugs in the long-term.
I've separated out these variables from the backup state, please see
the attached v9 patch.
I've also taken help of the error callback mechanism to clean up the
allocated memory in case of a failure. For do_pg_abort_backup() cases,
I think we don't need to clean the memory because that gets called on
proc exit (before_shmem_exit()).Memory could still bloat while the process running the SQL functions
is running depending on the error code path, anyway.
I didn't get your point. Can you please elaborate it? I think adding
error callbacks at the right places would free up the memory for us.
Please note that we already are causing memory leaks on HEAD today.
I addressed the above review comments. I also changed a wrong comment
[1]: * Essentially what this does is to create a backup label file in $PGDATA, * where it will be archived as part of the backup dump. The label file * contains the user-supplied label string (typically this would be used * to tell where the backup dump will be stored) and the starting time and * starting WAL location for the dump.
exclusive backup.
I'm attaching v9 patch set herewith, 0001 - refactors the backup code
with backup state, 0002 - adds error callbacks to clean up the memory
allocated for backup variables. Please review them further.
[1]: * Essentially what this does is to create a backup label file in $PGDATA, * where it will be archived as part of the backup dump. The label file * contains the user-supplied label string (typically this would be used * to tell where the backup dump will be stored) and the starting time and * starting WAL location for the dump.
* Essentially what this does is to create a backup label file in $PGDATA,
* where it will be archived as part of the backup dump. The label file
* contains the user-supplied label string (typically this would be used
* to tell where the backup dump will be stored) and the starting time and
* starting WAL location for the dump.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v9-0001-Refactor-backup-related-code.patchapplication/x-patch; name=v9-0001-Refactor-backup-related-code.patchDownload
From ab1e86ac7fb75a2d2219c7681ead40faf8c01446 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Tue, 20 Sep 2022 10:04:29 +0000
Subject: [PATCH v9] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 247 +++++++++---------------
src/backend/access/transam/xlogbackup.c | 151 +++++++++++++++
src/backend/access/transam/xlogfuncs.c | 90 +++++----
src/backend/backup/basebackup.c | 41 ++--
src/include/access/xlog.h | 9 +-
src/include/access/xlogbackup.h | 53 +++++
7 files changed, 384 insertions(+), 208 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index eb0430fe98..d25750e783 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,40 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
- *
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
- *
- * Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup state.
+ *
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster), and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.).
+ *
+ * The tablespace map contents are appended to passed-in parameter
+ * tablespace_map and the caller is responsible for including it in the backup
+ * archive as 'tablespace_map'. The tablespace_map file is required mainly for
+ * tar format in windows as native windows utilities are not able to create
+ * symlinks while extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
+ *
+ * It fills in backup state with information such as the minimum WAL location
+ * that must be present to restore from this backup (starttli), and the
+ * corresponding timeline ID (starttli), last checkpoint location
+ * (checkpointloc), start time of the backup (starttime), whether the backup started
+ * in recovery (started_in_recovery) and so on.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(BackupState *state, bool fast, List **tablespaces,
+ StringInfo tablespace_map)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8288,7 +8282,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("WAL level not sufficient for making an online backup"),
errhint("wal_level must be set to \"replica\" or \"logical\" at server start.")));
- if (strlen(backupidstr) > MAXPGPATH)
+ if (strlen(state->name) > MAXPGPATH)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
@@ -8385,9 +8379,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8398,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,19 +8430,16 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
- * Construct tablespace_map file.
+ * Construct tablespace_map contents.
*/
datadirpathlen = strlen(DataDir);
@@ -8525,46 +8516,23 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8564,43 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
- * Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * It fills in backup state with information such as the last WAL location that
+ * must be present to restore from this backup (stoppoint), and the
+ * corresponding timeline ID (stoptli), stop time of the backup (stoptime) and
+ * so on.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
+ *
+ * It is the responsibility of the caller to create backup label contents by
+ * calling build_backup_content() with appropriate parameters.
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8641,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
- */
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8681,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8693,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,24 +8705,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
{
+ StringInfo history_file;
+
/*
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,45 +8733,36 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Construct history file contents. */
+ history_file = makeStringInfo();
+ build_backup_content(state, true, history_file);
+
+ fprintf(fp, "%s", history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not write file \"%s\": %m",
histfilepath)));
+ pfree(history_file->data);
+
/*
* Clean out any no-longer-needed history files. As a side effect,
* this will post a .ready file for the newly created history file,
@@ -8855,15 +8794,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8844,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..38c37cc7db
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,151 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
+#include "utils/memutils.h"
+
+static void fill_backup_label_name(BackupState *state, const char* name);
+
+/*
+ * Allocate backup state and fill in the passed-in label name.
+ *
+ * Note that the caller has to ensure right memory context is set for the
+ * backup state.
+ */
+BackupState *
+allocate_backup_state(const char *name)
+{
+ BackupState *state;
+
+ state = (BackupState *) palloc0(sizeof(BackupState));
+
+ fill_backup_label_name(state, name);
+
+ return state;
+}
+
+/*
+ * Reset backup state and its members and fill in the passed-in label name.
+ */
+void
+reset_backup_state(BackupState *state, const char *name)
+{
+ if (state == NULL)
+ return; /* nothing to do */
+
+ MemSet(state, 0, sizeof(BackupState));
+ MemSet(state->name, '\0', sizeof(state->name));
+
+ fill_backup_label_name(state, name);
+}
+
+/*
+ * Thin helper function to fill label name (after truncating it, if necessary)
+ * in backup state.
+ */
+static void
+fill_backup_label_name(BackupState *state, const char* name)
+{
+ Size length;
+
+ length = strlen(name);
+
+ /*
+ * Truncate the backup label name whose length is greater than MAXPGPATH
+ * bytes as we will anyways error out later in do_pg_backup_start(). We
+ * truncate it to be one byte more to figure out the name has actually
+ * overflown.
+ */
+ if (length > MAXPGPATH)
+ length = MAXPGPATH + 1;
+
+ memcpy(state->name, name, length);
+}
+
+/*
+ * Deallocate backup state, backup_label and tablespace_map variables.
+ */
+void
+deallocate_backup_variables(BackupState *state, StringInfo backup_label,
+ StringInfo tablespace_map)
+{
+ if (state != NULL)
+ pfree(state);
+
+ if (backup_label != NULL)
+ pfree(backup_label->data);
+
+ if (tablespace_map != NULL)
+ pfree(tablespace_map->data);
+}
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile, StringInfo str)
+{
+ char startstrbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+
+ Assert(state != NULL);
+ Assert(str != NULL);
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ char stopstrfbuf[128];
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..e078579e6f 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -39,19 +40,15 @@
#include "utils/tuplestore.h"
/*
- * Store label file and tablespace map during backups.
+ * Backup related variables.
*/
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
+static StringInfo tablespace_map = NULL;
/*
- * pg_backup_start: set up for taking an on-line backup dump
+ * pg_backup_start: start taking an on-line backup.
*
- * Essentially what this does is to create a backup label file in $PGDATA,
- * where it will be archived as part of the backup dump. The label file
- * contains the user-supplied label string (typically this would be used
- * to tell where the backup dump will be stored) and the starting time and
- * starting WAL location for the dump.
+ * This function starts the backup and creates tablespace_map contents.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -62,7 +59,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +70,33 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state and tablespace_map need to be long-lived as they are used
+ * in pg_backup_stop(), hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+
+ if (backup_state == NULL)
+ backup_state = allocate_backup_state(backupidstr);
+ else
+ reset_backup_state(backup_state, backupidstr);
+
+ /*
+ * tablespace_map memory may have been enlarged previously, hence we free
+ * it up and allocate again instead of resetting to save some memory.
+ */
+ if (tablespace_map != NULL)
+ {
+ pfree(tablespace_map->data);
+ tablespace_map = NULL;
+ }
+
+ tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backup_state, fast, NULL, tablespace_map);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -98,6 +107,9 @@ pg_backup_start(PG_FUNCTION_ARGS)
* allows the user to choose if they want to wait for the WAL to be archived
* or if we should just return as soon as the WAL record is written.
*
+ * This function stops an in-progress backup, creates backup_label contents and
+ * it returns the backup stop LSN, backup_label and tablespace_map contents.
+ *
* Permission checking for this function is managed through the normal
* GRANT system.
*/
@@ -108,9 +120,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
+ StringInfo backup_label;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -123,23 +134,36 @@ pg_backup_stop(PG_FUNCTION_ARGS)
errmsg("backup is not in progress"),
errhint("Did you call pg_backup_start()?")));
+ /*
+ * Assert the fact that these backup variables are allocated in
+ * pg_backup_start().
+ */
+ Assert(backup_state != NULL);
+ Assert(tablespace_map != NULL);
+
/*
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /*
+ * Construct backup_label contents. Note that the backup_label doesn't have
+ * to be in TopMemoryContext unlike backup_state and tablespace_map as it
+ * is short-lived.
+ */
+ backup_label = makeStringInfo();
+ build_backup_content(backup_state, false, backup_label);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_label->data);
+ values[2] = CStringGetTextDatum(tablespace_map->data);
+
+ /* Deallocate backup related variables. */
+ deallocate_backup_variables(backup_state, backup_label, tablespace_map);
+ backup_state = NULL;
+ backup_label = NULL;
+ tablespace_map = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 1f1cff1a58..7ba4f340bc 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include <access/xlogbackup.h>
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
+ StringInfo backup_label;
+ StringInfo tablespace_map;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +250,25 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ /* Allocate backup related varilables. */
+ backup_state = allocate_backup_state(opt->label);
+ backup_label = makeStringInfo();
+ tablespace_map = makeStringInfo();
+
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(backup_state, opt->fastcheckpoint,
+ &state.tablespaces, tablespace_map);
+
+ /* Construct backup_label contents. */
+ build_backup_content(backup_state, false, backup_label);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -317,14 +326,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
- &manifest);
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_label->data, &manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
- &manifest);
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ tablespace_map->data, &manifest);
sendtblspclinks = false;
}
@@ -374,7 +383,13 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ deallocate_backup_variables(backup_state, backup_label,
+ tablespace_map);
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3dbfa6b593..5a47c5552b 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include <access/xlogbackup.h>
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,9 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(BackupState *state, bool fast,
+ List **tablespaces, StringInfo tablespace_map);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..5b43b7d1d7
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,53 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+
+ /*
+ * Null-terminated backup label name. If the label name is of size more
+ * than MAXPGPATH, an error is emitted in do_pg_backup_start(). We do not
+ * want to over-allocate memory in this structure, hence we truncate it to
+ * MAXPGPATH + 2 bytes (one extra byte for representing that the name has
+ * overflown, and another extra byte for null-termination).
+ */
+ char name[MAXPGPATH + 2];
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupState;
+
+extern BackupState *allocate_backup_state(const char *name);
+extern void reset_backup_state(BackupState *state, const char *name);
+extern void deallocate_backup_variables(BackupState *state,
+ StringInfo backup_label,
+ StringInfo tablespace_map);
+extern void build_backup_content(BackupState *state,
+ bool ishistoryfile,
+ StringInfo str);
+#endif /* XLOG_BACKUP_H */
--
2.34.1
v9-0002-Add-error-callbacks-for-deallocating-backup-relat.patchapplication/x-patch; name=v9-0002-Add-error-callbacks-for-deallocating-backup-relat.patchDownload
From b20d5455f746ec8f2ca4bd325f8636aa2b991438 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Tue, 20 Sep 2022 10:44:42 +0000
Subject: [PATCH v9] Add error callbacks for deallocating backup related
variables
This adds error callbacks for deallocating backup related
variables to save on some palloc-ed memory upon errors while
taking backup.
---
src/backend/access/transam/xlogbackup.c | 22 ++++++++++
src/backend/access/transam/xlogfuncs.c | 54 +++++++++++++++++++++++++
src/backend/backup/basebackup.c | 14 +++++++
src/backend/utils/error/elog.c | 17 ++++++++
src/include/access/xlogbackup.h | 11 +++++
src/include/utils/elog.h | 1 +
6 files changed, 119 insertions(+)
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
index 38c37cc7db..39210af41b 100644
--- a/src/backend/access/transam/xlogbackup.c
+++ b/src/backend/access/transam/xlogbackup.c
@@ -92,6 +92,28 @@ deallocate_backup_variables(BackupState *state, StringInfo backup_label,
pfree(tablespace_map->data);
}
+/*
+ * Erro callback for deallocating backup variables.
+ */
+void
+deallocate_backup_variables_err_cb(void *arg)
+{
+ BackupErrorInfo *errinfo = (BackupErrorInfo *) arg;
+
+ /*
+ * Since error callbacks are called for elevel lesser than ERROR, we
+ * proceed further only if elevel is ERROR or greater.
+ */
+ if (geterrlevel() < ERROR)
+ return;
+
+ if (errinfo == NULL)
+ return; /* nothing to do */
+
+ deallocate_backup_variables(errinfo->state, errinfo->backup_label,
+ errinfo->tablespace_map);
+}
+
/*
* Construct backup_label file or history file contents
*
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index e078579e6f..7d801aefc7 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -45,6 +45,32 @@
static BackupState *backup_state = NULL;
static StringInfo tablespace_map = NULL;
+/*
+ * Erro callback for deallocating and resetting backup variables.
+ */
+static void
+deallocate_and_reset_backup_variables_err_cb(void *arg)
+{
+ BackupErrorInfo *errinfo = (BackupErrorInfo *) arg;
+
+ /*
+ * Since error callbacks are called for elevel lesser than ERROR, we
+ * proceed further only if elevel is ERROR or greater.
+ */
+ if (geterrlevel() < ERROR)
+ return;
+
+ if (errinfo == NULL)
+ return; /* nothing to do */
+
+ deallocate_backup_variables(errinfo->state, errinfo->backup_label,
+ errinfo->tablespace_map);
+
+ /* Reset the static variables. */
+ backup_state = NULL;
+ tablespace_map = NULL;
+}
+
/*
* pg_backup_start: start taking an on-line backup.
*
@@ -61,6 +87,8 @@ pg_backup_start(PG_FUNCTION_ARGS)
char *backupidstr;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
backupidstr = text_to_cstring(backupid);
@@ -93,9 +121,21 @@ pg_backup_start(PG_FUNCTION_ARGS)
tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_and_reset_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = NULL;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
register_persistent_abort_backup_handler();
do_pg_backup_start(backup_state, fast, NULL, tablespace_map);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
PG_RETURN_LSN(backup_state->startpoint);
}
@@ -123,6 +163,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
bool waitforarchive = PG_GETARG_BOOL(0);
StringInfo backup_label;
SessionBackupState status = get_backup_status();
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
/* Initialize attributes information in the tuple descriptor */
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -141,12 +183,24 @@ pg_backup_stop(PG_FUNCTION_ARGS)
Assert(backup_state != NULL);
Assert(tablespace_map != NULL);
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_and_reset_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = NULL;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/*
* Stop the backup. Return a copy of the backup label and tablespace map
* so they can be written to disk by the caller.
*/
do_pg_backup_stop(backup_state, waitforarchive);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
/*
* Construct backup_label contents. Note that the backup_label doesn't have
* to be in TopMemoryContext unlike backup_state and tablespace_map as it
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 7ba4f340bc..3edbe4c460 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -236,6 +236,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
BackupState *backup_state;
StringInfo backup_label;
StringInfo tablespace_map;
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -260,6 +262,15 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_label = makeStringInfo();
tablespace_map = makeStringInfo();
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = backup_label;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
basebackup_progress_wait_checkpoint();
do_pg_backup_start(backup_state, opt->fastcheckpoint,
&state.tablespaces, tablespace_map);
@@ -663,6 +674,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
/* clean up the resource owner we created */
WalSndResourceCleanup(true);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
basebackup_progress_done();
}
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index 96c694da8f..d2868648db 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -1404,6 +1404,23 @@ geterrcode(void)
return edata->sqlerrcode;
}
+/*
+ * geterrlevel --- return the currently error level
+ *
+ * This is only intended for use in error callback subroutines, since there
+ * is no other place outside elog.c where the concept is meaningful.
+ */
+int
+geterrlevel(void)
+{
+ ErrorData *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ return edata->elevel;
+}
+
/*
* geterrposition --- return the currently set error position (0 if none)
*
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
index 5b43b7d1d7..e98fe6132c 100644
--- a/src/include/access/xlogbackup.h
+++ b/src/include/access/xlogbackup.h
@@ -42,6 +42,16 @@ typedef struct BackupState
pg_time_t stoptime; /* backup stop time */
} BackupState;
+/*
+ * Structure to hold input parameters for backup error callback.
+ */
+typedef struct BackupErrorInfo
+{
+ BackupState *state;
+ StringInfo backup_label;
+ StringInfo tablespace_map;
+} BackupErrorInfo;
+
extern BackupState *allocate_backup_state(const char *name);
extern void reset_backup_state(BackupState *state, const char *name);
extern void deallocate_backup_variables(BackupState *state,
@@ -50,4 +60,5 @@ extern void deallocate_backup_variables(BackupState *state,
extern void build_backup_content(BackupState *state,
bool ishistoryfile,
StringInfo str);
+extern void deallocate_backup_variables_err_cb(void *arg);
#endif /* XLOG_BACKUP_H */
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index 4dd9658a3c..a8955f9572 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -222,6 +222,7 @@ extern int internalerrquery(const char *query);
extern int err_generic_string(int field, const char *str);
extern int geterrcode(void);
+extern int geterrlevel(void);
extern int geterrposition(void);
extern int getinternalerrposition(void);
--
2.34.1
On 2022/09/20 20:43, Bharath Rupireddy wrote:
- if (strlen(backupidstr) > MAXPGPATH)
+ if (state->name_overflowed == true)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("backup label too long (max %d bytes)",
It does not strike me as a huge issue to force a truncation of such
backup label names. 1024 is large enough for basically all users,
in my opinion. Or you could just truncate in the internal logic, but
still complain at MAXPGPATH - 1 as the last byte would be for the zero
termination. In short, there is no need to complicate things with
name_overflowed.We currently allow MAXPGPATH bytes of label name excluding null
termination. I don't want to change this behaviour. In the attached v9
patch, I'm truncating the larger names to MAXPGPATH + 1 bytes in
backup state (one extra byte for representing that the name has
overflown, and another extra byte for null termination).
This looks much complicated to me.
Instead of making allocate_backup_state() or reset_backup_state()
store the label name in BackupState before do_pg_backup_start(),
how about making do_pg_backup_start() do that after checking its length?
Seems this can simplify the code very much.
If so, ISTM that we can replace allocate_backup_state() and
reset_backup_state() with just palloc0() and MemSet(), respectively.
Also we can remove fill_backup_label_name().
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On Tue, Sep 20, 2022 at 05:13:49PM +0530, Bharath Rupireddy wrote:
On Tue, Sep 20, 2022 at 7:20 AM Michael Paquier <michael@paquier.xyz> wrote:
I've separated out these variables from the backup state, please see
the attached v9 patch.
Thanks, the separation looks cleaner.
I've also taken help of the error callback mechanism to clean up the
allocated memory in case of a failure. For do_pg_abort_backup() cases,
I think we don't need to clean the memory because that gets called on
proc exit (before_shmem_exit()).Memory could still bloat while the process running the SQL functions
is running depending on the error code path, anyway.I didn't get your point. Can you please elaborate it? I think adding
error callbacks at the right places would free up the memory for us.
Please note that we already are causing memory leaks on HEAD today.
I mean that HEAD makes no effort in freeing this memory in
TopMemoryContext on session ERROR.
I addressed the above review comments. I also changed a wrong comment
[1] that lies before pg_backup_start() even after the removal of
exclusive backup.I'm attaching v9 patch set herewith, 0001 - refactors the backup code
with backup state, 0002 - adds error callbacks to clean up the memory
allocated for backup variables. Please review them further.
I have a few comments on 0001.
+#include <access/xlogbackup.h>
Double quotes wanted here.
deallocate_backup_variables() is the only part of xlogbackup.c that
includes references of the tablespace map_and backup_label
StringInfos. I would be tempted to fully decouple that from
xlogbackup.c/h for the time being.
- tblspc_map_file = makeStringInfo();
Not sure that there is a need for a rename here.
+void
+build_backup_content(BackupState *state, bool ishistoryfile, StringInfo str)
+{
It would be more natural to have build_backup_content() do by itself
the initialization of the StringInfo for the contents of backup_label
and return it as a result of the routine? This is short-lived in
xlogfuncs.c when the backup ends.
@@ -248,18 +250,25 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
[...]
+ /* Construct backup_label contents. */
+ build_backup_content(backup_state, false, backup_label);
Actually, for base backups, perhaps it would be more intuitive to
build and free the StringInfo of the backup_label when we send it for
base.tar rather than initializing it at the beginning and freeing it
at the end?
- * pg_backup_start: set up for taking an on-line backup dump
+ * pg_backup_start: start taking an on-line backup.
*
- * Essentially what this does is to create a backup label file in $PGDATA,
- * where it will be archived as part of the backup dump. The label file
- * contains the user-supplied label string (typically this would be used
- * to tell where the backup dump will be stored) and the starting time and
- * starting WAL location for the dump.
+ * This function starts the backup and creates tablespace_map contents.
The last part of the comment is still correct while the former is not,
so this loses some information.
--
Michael
On Wed, Sep 21, 2022 at 03:45:49PM +0900, Fujii Masao wrote:
Instead of making allocate_backup_state() or reset_backup_state()
store the label name in BackupState before do_pg_backup_start(),
how about making do_pg_backup_start() do that after checking its length?
Seems this can simplify the code very much.If so, ISTM that we can replace allocate_backup_state() and
reset_backup_state() with just palloc0() and MemSet(), respectively.
Also we can remove fill_backup_label_name().
Yep, agreed. Having all these routines feels a bit overengineered.
--
Michael
On Wed, Sep 21, 2022 at 12:15 PM Fujii Masao
<masao.fujii@oss.nttdata.com> wrote:
This looks much complicated to me.
Instead of making allocate_backup_state() or reset_backup_state()
store the label name in BackupState before do_pg_backup_start(),
how about making do_pg_backup_start() do that after checking its length?
Seems this can simplify the code very much.If so, ISTM that we can replace allocate_backup_state() and
reset_backup_state() with just palloc0() and MemSet(), respectively.
Also we can remove fill_backup_label_name().
Yes, that makes things simpler. I will post the v10 patch-set soon.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Wed, Sep 21, 2022 at 12:27 PM Michael Paquier <michael@paquier.xyz> wrote:
I've also taken help of the error callback mechanism to clean up the
allocated memory in case of a failure. For do_pg_abort_backup() cases,
I think we don't need to clean the memory because that gets called on
proc exit (before_shmem_exit()).Memory could still bloat while the process running the SQL functions
is running depending on the error code path, anyway.I didn't get your point. Can you please elaborate it? I think adding
error callbacks at the right places would free up the memory for us.
Please note that we already are causing memory leaks on HEAD today.I mean that HEAD makes no effort in freeing this memory in
TopMemoryContext on session ERROR.
Correct. We can also solve it as part of this commit. Please let me
know your thoughts on 0002 patch.
I have a few comments on 0001.
+#include <access/xlogbackup.h>
Double quotes wanted here.
Ah, my bad. Corrected now.
deallocate_backup_variables() is the only part of xlogbackup.c that
includes references of the tablespace map_and backup_label
StringInfos. I would be tempted to fully decouple that from
xlogbackup.c/h for the time being.
There's no problem with it IMO, after all, they are backup related
variables. And that function reduces a bit of duplicate code.
- tblspc_map_file = makeStringInfo();
Not sure that there is a need for a rename here.
We're referring tablespace_map and backup_label internally all around,
just to be in sync, I wanted to rename it while we're refactoring this
code.
+void +build_backup_content(BackupState *state, bool ishistoryfile, StringInfo str) +{ It would be more natural to have build_backup_content() do by itself the initialization of the StringInfo for the contents of backup_label and return it as a result of the routine? This is short-lived in xlogfuncs.c when the backup ends.
See the below explaination.
@@ -248,18 +250,25 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) [...] + /* Construct backup_label contents. */ + build_backup_content(backup_state, false, backup_label);Actually, for base backups, perhaps it would be more intuitive to
build and free the StringInfo of the backup_label when we send it for
base.tar rather than initializing it at the beginning and freeing it
at the end?
sendFileWithContent() is in a for-loop and we are good if we call
build_backup_content() before do_pg_backup_start() just once. Also,
allocating backup_label in the for loop makes error handling trickier,
how do we free-up when sendFileWithContent() errors out? Of course, we
can allocate backup_label once even in the for loop with bool
first_time sort of variable and store StringInfo *ptr_backup_label; in
error callback info, but that would make things unnecessarily complex,
instead we're good allocating and creating backup_label content at the
beginning and freeing-it up at the end.
- * pg_backup_start: set up for taking an on-line backup dump + * pg_backup_start: start taking an on-line backup. * - * Essentially what this does is to create a backup label file in $PGDATA, - * where it will be archived as part of the backup dump. The label file - * contains the user-supplied label string (typically this would be used - * to tell where the backup dump will be stored) and the starting time and - * starting WAL location for the dump. + * This function starts the backup and creates tablespace_map contents.The last part of the comment is still correct while the former is not,
so this loses some information.
Added that part before pg_backup_stop() now where it makes sense with
the refactoring.
I'm attaching the v10 patch-set with the above review comments addressed.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v10-0001-Refactor-backup-related-code.patchapplication/x-patch; name=v10-0001-Refactor-backup-related-code.patchDownload
From cb93211220d73cfb4ae832ff4e04fd46e706ddfe Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Wed, 21 Sep 2022 10:52:06 +0000
Subject: [PATCH v10] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 246 +++++++++---------------
src/backend/access/transam/xlogbackup.c | 96 +++++++++
src/backend/access/transam/xlogfuncs.c | 108 +++++++----
src/backend/backup/basebackup.c | 45 +++--
src/include/access/xlog.h | 10 +-
src/include/access/xlogbackup.h | 48 +++++
7 files changed, 345 insertions(+), 209 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e028124672..c44b0ffd43 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8236,46 +8236,40 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
- *
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
- *
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
- *
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
- *
- * Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * backup state.
+ *
+ * Input parameters are "state" (containing backup state), "fast" (if true,
+ * we do the checkpoint in immediate mode to make it faster), and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.).
+ *
+ * The tablespace map contents are appended to passed-in parameter
+ * tablespace_map and the caller is responsible for including it in the backup
+ * archive as 'tablespace_map'. The tablespace_map file is required mainly for
+ * tar format in windows as native windows utilities are not able to create
+ * symlinks while extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
+ *
+ * It fills in backup state with information such as the minimum WAL location
+ * that must be present to restore from this backup (starttli), and the
+ * corresponding timeline ID (starttli), last checkpoint location
+ * (checkpointloc), start time of the backup (starttime), whether the backup
+ * started in recovery (started_in_recovery) and so on.
*
* Every successfully started backup must be stopped by calling
- * do_pg_backup_stop() or do_pg_abort_backup(). There can be many
- * backups active at the same time.
+ * do_pg_backup_stop() or do_pg_abort_backup(). There can be many backups
+ * active at the same time.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
+ BackupState *state, StringInfo tablespace_map)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8294,6 +8288,8 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("backup label too long (max %d bytes)",
MAXPGPATH)));
+ memcpy(state->name, backupidstr, strlen(backupidstr));
+
/*
* Mark backup active in shared memory. We must do full-page WAL writes
* during an on-line backup even if not doing so at other times, because
@@ -8385,9 +8381,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8400,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,19 +8432,16 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
- * Construct tablespace_map file.
+ * Construct tablespace_map contents.
*/
datadirpathlen = strlen(DataDir);
@@ -8525,46 +8518,23 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
if (tablespaces)
*tablespaces = lappend(*tablespaces, ti);
- appendStringInfo(tblspcmapfile, "%s %s\n",
+ appendStringInfo(tablespace_map, "%s %s\n",
ti->oid, escapedpath.data);
pfree(escapedpath.data);
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8596,48 +8566,43 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
- * Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * It fills in backup state with information such as the last WAL location that
+ * must be present to restore from this backup (stoppoint), and the
+ * corresponding timeline ID (stoptli), stop time of the backup (stoptime) and
+ * so on.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
+ *
+ * It is the responsibility of the caller to create backup label contents by
+ * calling build_backup_content() with appropriate parameters.
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8678,29 +8643,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
- */
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8736,7 +8683,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8748,7 +8695,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8760,24 +8707,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
{
+ StringInfo history_file;
+
/*
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8785,39 +8735,29 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Construct history file contents. */
+ history_file = makeStringInfo();
+ build_backup_content(state, true, history_file);
+
+ fprintf(fp, "%s", history_file->data);
+ pfree(history_file->data);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8855,15 +8795,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8904,13 +8845,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..a9addf6e0c
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,96 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.c
+ *
+ * This file contains backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
+#include "utils/memutils.h"
+
+/*
+ * Construct backup_label file or history file contents
+ *
+ * When ishistoryfile is true, it creates contents for backup history file,
+ * otherwise, it creates contents for backup_label file.
+ *
+ * It is the caller's responsibility to pass-in an allocated string 'str' for
+ * the output.
+ */
+void
+build_backup_content(BackupState *state, bool ishistoryfile, StringInfo str)
+{
+ char startstrbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+
+ Assert(state != NULL);
+ Assert(str != NULL);
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(str, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(str, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(str, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(str, "BACKUP METHOD: streamed\n");
+ appendStringInfo(str, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(str, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(str, "LABEL: %s\n", state->name);
+ appendStringInfo(str, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ char stopstrfbuf[128];
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(str, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(str, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+}
+
+/*
+ * Deallocate backup state, backup_label and tablespace_map variables.
+ */
+void
+deallocate_backup_variables(BackupState *state, StringInfo backup_label,
+ StringInfo tablespace_map)
+{
+ if (state != NULL)
+ pfree(state);
+
+ if (backup_label != NULL)
+ pfree(backup_label->data);
+
+ if (tablespace_map != NULL)
+ pfree(tablespace_map->data);
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..8b06d977b9 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -39,19 +40,15 @@
#include "utils/tuplestore.h"
/*
- * Store label file and tablespace map during backups.
+ * Backup related variables.
*/
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
+static StringInfo tablespace_map = NULL;
/*
- * pg_backup_start: set up for taking an on-line backup dump
+ * pg_backup_start: start taking an on-line backup.
*
- * Essentially what this does is to create a backup label file in $PGDATA,
- * where it will be archived as part of the backup dump. The label file
- * contains the user-supplied label string (typically this would be used
- * to tell where the backup dump will be stored) and the starting time and
- * starting WAL location for the dump.
+ * This function starts the backup and creates tablespace_map contents.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -62,7 +59,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +70,42 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state and tablespace_map need to be long-lived as they are used
+ * in pg_backup_stop(), hence allocate in TopMemoryContext.
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+
+ if (backup_state == NULL)
+ {
+ /* Allocate backup state, if not done yet. */
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ }
+ else
+ {
+ /*
+ * Reset backup state and its members, if it was previously allocated.
+ */
+ MemSet(backup_state, 0, sizeof(BackupState));
+ MemSet(backup_state->name, '\0', sizeof(backup_state->name));
+ }
+
+ /*
+ * tablespace_map memory may have been enlarged previously, hence we free
+ * it up and allocate again instead of resetting, to save some memory.
+ */
+ if (tablespace_map != NULL)
+ {
+ pfree(tablespace_map->data);
+ tablespace_map = NULL;
+ }
+
+ tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -98,6 +116,15 @@ pg_backup_start(PG_FUNCTION_ARGS)
* allows the user to choose if they want to wait for the WAL to be archived
* or if we should just return as soon as the WAL record is written.
*
+ * This function stops an in-progress backup, creates backup_label contents and
+ * it returns the backup stop LSN, backup_label and tablespace_map contents.
+ *
+ * The backup_label contains the user-supplied label string (typically this
+ * would be used to tell where the backup dump will be stored), the starting
+ * time, starting WAL location for the dump and so on. It is the caller's
+ * responsibility to write backup_label and tablespace_map contents as separate
+ * files to disk.
+ *
* Permission checking for this function is managed through the normal
* GRANT system.
*/
@@ -108,9 +135,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
+ StringInfo backup_label;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -124,22 +150,34 @@ pg_backup_stop(PG_FUNCTION_ARGS)
errhint("Did you call pg_backup_start()?")));
/*
- * Stop the backup. Return a copy of the backup label and tablespace map
- * so they can be written to disk by the caller.
+ * Assert the fact that these backup variables are allocated in
+ * pg_backup_start().
+ */
+ Assert(backup_state != NULL);
+ Assert(tablespace_map != NULL);
+
+ /*
+ * Stop the backup.
+ */
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /*
+ * Construct backup_label contents. Note that the backup_label doesn't have
+ * to be in TopMemoryContext unlike backup_state and tablespace_map as it
+ * is short-lived.
*/
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ backup_label = makeStringInfo();
+ build_backup_content(backup_state, false, backup_label);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_label->data);
+ values[2] = CStringGetTextDatum(tablespace_map->data);
+
+ /* Deallocate backup related variables. */
+ deallocate_backup_variables(backup_state, backup_label, tablespace_map);
+ backup_state = NULL;
+ backup_label = NULL;
+ tablespace_map = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index dd103a8689..fb3b59832b 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,10 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
+ StringInfo backup_label;
+ StringInfo tablespace_map;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +250,25 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ /* Allocate backup related varilables. */
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ backup_label = makeStringInfo();
+ tablespace_map = makeStringInfo();
+
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(opt->label, opt->fastcheckpoint, &state.tablespaces,
+ backup_state, tablespace_map);
+
+ /* Construct backup_label contents. */
+ build_backup_content(backup_state, false, backup_label);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -317,14 +326,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_begin_archive(sink, "base.tar");
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
- &manifest);
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_label->data, &manifest);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
- &manifest);
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ tablespace_map->data, &manifest);
sendtblspclinks = false;
}
@@ -374,7 +383,17 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ /* Deallocate backup related variables. */
+ deallocate_backup_variables(backup_state, backup_label,
+ tablespace_map);
+ backup_state = NULL;
+ backup_label = NULL;
+ tablespace_map = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3dbfa6b593..4e9aa617a0 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlogbackup.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,10 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(const char *backupidstr, bool fast,
+ List **tablespaces, BackupState *state,
+ StringInfo tablespace_map);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..69b960c417
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,48 @@
+/*-------------------------------------------------------------------------
+ * xlogbackup.h
+ *
+ * Declarations for the backup related code.
+ *
+ * Copyright (c) 2022, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Following are the fields captured when the backup starts. */
+
+ /*
+ * Backup label name with the extra byte for null-termination.
+ */
+ char name[MAXPGPATH + 1];
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Following are the fields captured during or after the backup stops. */
+
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupState;
+
+extern void build_backup_content(BackupState *state,
+ bool ishistoryfile,
+ StringInfo str);
+extern void deallocate_backup_variables(BackupState *state,
+ StringInfo backup_label,
+ StringInfo tablespace_map);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
v10-0002-Add-error-callbacks-for-deallocating-backup-rela.patchapplication/x-patch; name=v10-0002-Add-error-callbacks-for-deallocating-backup-rela.patchDownload
From 9e685c75676c763ade9c0b2d938d1b2e567e2c93 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Wed, 21 Sep 2022 11:02:53 +0000
Subject: [PATCH v10] Add error callbacks for deallocating backup related
variables
This adds error callbacks for deallocating backup related
variables to avoid memory bloat and save on some memory upon
errors while taking backup.
---
src/backend/access/transam/xlogbackup.c | 22 ++++++++++
src/backend/access/transam/xlogfuncs.c | 54 +++++++++++++++++++++++++
src/backend/backup/basebackup.c | 14 +++++++
src/backend/utils/error/elog.c | 17 ++++++++
src/include/access/xlogbackup.h | 11 +++++
src/include/utils/elog.h | 1 +
6 files changed, 119 insertions(+)
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
index a9addf6e0c..83724548ef 100644
--- a/src/backend/access/transam/xlogbackup.c
+++ b/src/backend/access/transam/xlogbackup.c
@@ -94,3 +94,25 @@ deallocate_backup_variables(BackupState *state, StringInfo backup_label,
if (tablespace_map != NULL)
pfree(tablespace_map->data);
}
+
+/*
+ * Error callback for deallocating backup variables.
+ */
+void
+deallocate_backup_variables_err_cb(void *arg)
+{
+ BackupErrorInfo *errinfo = (BackupErrorInfo *) arg;
+
+ /*
+ * Since error callbacks are called for elevel lesser than ERROR, we
+ * proceed further only if elevel is ERROR or greater.
+ */
+ if (geterrlevel() < ERROR)
+ return;
+
+ if (errinfo == NULL)
+ return; /* nothing to do */
+
+ deallocate_backup_variables(errinfo->state, errinfo->backup_label,
+ errinfo->tablespace_map);
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 8b06d977b9..49a88a9068 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -45,6 +45,32 @@
static BackupState *backup_state = NULL;
static StringInfo tablespace_map = NULL;
+/*
+ * Erro callback for deallocating and resetting backup variables.
+ */
+static void
+deallocate_and_reset_backup_variables_err_cb(void *arg)
+{
+ BackupErrorInfo *errinfo = (BackupErrorInfo *) arg;
+
+ /*
+ * Since error callbacks are called for elevel lesser than ERROR, we
+ * proceed further only if elevel is ERROR or greater.
+ */
+ if (geterrlevel() < ERROR)
+ return;
+
+ if (errinfo == NULL)
+ return; /* nothing to do */
+
+ deallocate_backup_variables(errinfo->state, errinfo->backup_label,
+ errinfo->tablespace_map);
+
+ /* Reset the static variables. */
+ backup_state = NULL;
+ tablespace_map = NULL;
+}
+
/*
* pg_backup_start: start taking an on-line backup.
*
@@ -61,6 +87,8 @@ pg_backup_start(PG_FUNCTION_ARGS)
char *backupidstr;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
backupidstr = text_to_cstring(backupid);
@@ -102,9 +130,21 @@ pg_backup_start(PG_FUNCTION_ARGS)
tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_and_reset_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = NULL;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
register_persistent_abort_backup_handler();
do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
PG_RETURN_LSN(backup_state->startpoint);
}
@@ -138,6 +178,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
bool waitforarchive = PG_GETARG_BOOL(0);
StringInfo backup_label;
SessionBackupState status = get_backup_status();
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
/* Initialize attributes information in the tuple descriptor */
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -156,6 +198,15 @@ pg_backup_stop(PG_FUNCTION_ARGS)
Assert(backup_state != NULL);
Assert(tablespace_map != NULL);
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_and_reset_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = NULL;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
/*
* Stop the backup.
*/
@@ -169,6 +220,9 @@ pg_backup_stop(PG_FUNCTION_ARGS)
backup_label = makeStringInfo();
build_backup_content(backup_state, false, backup_label);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
values[0] = LSNGetDatum(backup_state->stoppoint);
values[1] = CStringGetTextDatum(backup_label->data);
values[2] = CStringGetTextDatum(tablespace_map->data);
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index fb3b59832b..d20aa20950 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -236,6 +236,8 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
BackupState *backup_state;
StringInfo backup_label;
StringInfo tablespace_map;
+ ErrorContextCallback errcallback;
+ BackupErrorInfo errinfo;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -260,6 +262,15 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_label = makeStringInfo();
tablespace_map = makeStringInfo();
+ /* Set up callback to clean up backup related variables */
+ errcallback.callback = deallocate_backup_variables_err_cb;
+ errinfo.state = backup_state;
+ errinfo.backup_label = backup_label;
+ errinfo.tablespace_map = tablespace_map;
+ errcallback.arg = (void *) &errinfo;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
basebackup_progress_wait_checkpoint();
do_pg_backup_start(opt->label, opt->fastcheckpoint, &state.tablespaces,
backup_state, tablespace_map);
@@ -667,6 +678,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
/* clean up the resource owner we created */
WalSndResourceCleanup(true);
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+
basebackup_progress_done();
}
diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c
index eb724a9d7f..6519fb310c 100644
--- a/src/backend/utils/error/elog.c
+++ b/src/backend/utils/error/elog.c
@@ -1404,6 +1404,23 @@ geterrcode(void)
return edata->sqlerrcode;
}
+/*
+ * geterrlevel --- return the currently error level
+ *
+ * This is only intended for use in error callback subroutines, since there
+ * is no other place outside elog.c where the concept is meaningful.
+ */
+int
+geterrlevel(void)
+{
+ ErrorData *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ return edata->elevel;
+}
+
/*
* geterrposition --- return the currently set error position (0 if none)
*
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
index 69b960c417..4c80070ce6 100644
--- a/src/include/access/xlogbackup.h
+++ b/src/include/access/xlogbackup.h
@@ -38,11 +38,22 @@ typedef struct BackupState
pg_time_t stoptime; /* backup stop time */
} BackupState;
+/*
+ * Structure to hold input parameters for backup error callback.
+ */
+typedef struct BackupErrorInfo
+{
+ BackupState *state;
+ StringInfo backup_label;
+ StringInfo tablespace_map;
+} BackupErrorInfo;
+
extern void build_backup_content(BackupState *state,
bool ishistoryfile,
StringInfo str);
extern void deallocate_backup_variables(BackupState *state,
StringInfo backup_label,
StringInfo tablespace_map);
+extern void deallocate_backup_variables_err_cb(void *arg);
#endif /* XLOG_BACKUP_H */
diff --git a/src/include/utils/elog.h b/src/include/utils/elog.h
index 4dd9658a3c..a8955f9572 100644
--- a/src/include/utils/elog.h
+++ b/src/include/utils/elog.h
@@ -222,6 +222,7 @@ extern int internalerrquery(const char *query);
extern int err_generic_string(int field, const char *str);
extern int geterrcode(void);
+extern int geterrlevel(void);
extern int geterrposition(void);
extern int getinternalerrposition(void);
--
2.34.1
On Wed, Sep 21, 2022 at 05:10:39PM +0530, Bharath Rupireddy wrote:
deallocate_backup_variables() is the only part of xlogbackup.c that
includes references of the tablespace map_and backup_label
StringInfos. I would be tempted to fully decouple that from
xlogbackup.c/h for the time being.There's no problem with it IMO, after all, they are backup related
variables. And that function reduces a bit of duplicate code.
Hmm. I'd like to disagree with this statement :)
Added that part before pg_backup_stop() now where it makes sense with
the refactoring.
I have put my hands on 0001, and finished with the attached, that
includes many fixes and tweaks. Some of the variable renames felt out
of place, while some comments were overly verbose for their purpose,
though for the last part we did not lose any information in the last
version proposed.
As I suspected, the deallocate routine felt unnecessary, as
xlogbackup.c/h have no idea what these are. The remark is
particularly true for the StringInfo of the backup_label file: for
basebackup.c we need to build it when sending base.tar and in
xlogfuncs.c we need it only at the backup stop phase. The code was
actually a bit wrong, because free-ing StringInfos requires to free
its ->data and then the main object (stringinfo.h explains that). My
tweaks have shaved something like 10%~15% of the patch, while making
it IMO more readable.
A second issue I had was with the build function, and again it seemed
much cleaner to let the routine do the makeStringInfo() and return the
result. This is not the most popular routine ever, but this reduces
the workload of the caller of build_backup_content().
So, opinions?
--
Michael
Attachments:
v11-0001-Refactor-backup-related-code.patchtext/x-diff; charset=us-asciiDownload
From 22216c4b6b75607d45e49620264b1af606396bd4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Thu, 22 Sep 2022 16:41:55 +0900
Subject: [PATCH v11] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/include/access/xlog.h | 10 +-
src/include/access/xlogbackup.h | 42 +++++
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 224 ++++++++----------------
src/backend/access/transam/xlogbackup.c | 81 +++++++++
src/backend/access/transam/xlogfuncs.c | 96 ++++++----
src/backend/backup/basebackup.c | 44 +++--
7 files changed, 298 insertions(+), 200 deletions(-)
create mode 100644 src/include/access/xlogbackup.h
create mode 100644 src/backend/access/transam/xlogbackup.c
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3dbfa6b593..dce265098e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlogbackup.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,10 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(const char *backupidstr, bool fast,
+ List **tablespaces, BackupState *state,
+ StringInfo tblspcmapfile);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..ccdfefe117
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,42 @@
+/*-------------------------------------------------------------------------
+ *
+ * xlogbackup.h
+ * Definitions for internals of base backups.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Fields saved at backup start */
+ /* Backup label name one extra byte for null-termination */
+ char name[MAXPGPATH + 1];
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Fields saved at the end of backup */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupState;
+
+extern StringInfo build_backup_content(BackupState *state,
+ bool ishistoryfile);
+
+#endif /* XLOG_BACKUP_H */
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f32b2124e6..e061554363 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8242,24 +8242,23 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state and tablespace map.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (the backup state), "fast" (if true, we do
+ * the checkpoint in immediate mode to make it faster), and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.).
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * The tablespace map contents are appended to passed-in parameter
+ * tablespace_map and the caller is responsible for including it in the backup
+ * archive as 'tablespace_map'. The tablespace_map file is required mainly for
+ * tar format in windows as native windows utilities are not able to create
+ * symlinks while extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
- *
- * Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * It fills in backup_state with the information required for the backup,
+ * such as the minimum WAL location that must be present to restore from
+ * this backup (starttli) and the corresponding timeline ID (starttli).
*
* Every successfully started backup must be stopped by calling
* do_pg_backup_stop() or do_pg_abort_backup(). There can be many
@@ -8268,20 +8267,13 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
+ BackupState *state, StringInfo tblspcmapfile)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8300,6 +8292,8 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("backup label too long (max %d bytes)",
MAXPGPATH)));
+ memcpy(state->name, backupidstr, strlen(backupidstr));
+
/*
* Mark backup active in shared memory. We must do full-page WAL writes
* during an on-line backup even if not doing so at other times, because
@@ -8391,9 +8385,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8410,7 +8404,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8442,17 +8436,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
* Construct tablespace_map file.
*/
@@ -8538,39 +8529,16 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8602,48 +8570,38 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
- * Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup_state is filled with the information necessary to restore from this
+ * backup with its stop LSN (stoppoint), its timeline ID (stoptli), etc.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8684,29 +8642,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8742,7 +8682,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8754,7 +8694,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8766,24 +8706,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
{
+ StringInfo history_file;
+
/*
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8791,39 +8734,28 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Build and save the contents of the backup history file */
+ history_file = build_backup_content(state, true);
+ fprintf(fp, "%s", history_file->data);
+ pfree(history_file->data);
+ pfree(history_file);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8861,15 +8793,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8910,13 +8843,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..5b3a5cce73
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,81 @@
+/*-------------------------------------------------------------------------
+ *
+ * xlogbackup.c
+ * Internal routines for base backups.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
+#include "utils/memutils.h"
+
+/*
+ * Build contents for backup_label or backup history file.
+ *
+ * When ishistoryfile is true, it creates the contents for a backup history
+ * file, otherwise it creates contents for a backup_label file.
+ *
+ * Returns the result generated as a palloc'd StringInfo.
+ */
+StringInfo
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ char startstrbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+ StringInfo result = makeStringInfo();
+
+ Assert(state != NULL);
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
+ }
+
+ appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(result, "BACKUP METHOD: streamed\n");
+ appendStringInfo(result, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(result, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(result, "LABEL: %s\n", state->name);
+ appendStringInfo(result, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ char stopstrfbuf[128];
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+
+ return result;
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..7925ceeaad 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -39,19 +40,16 @@
#include "utils/tuplestore.h"
/*
- * Store label file and tablespace map during backups.
+ * Backup-related variables.
*/
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
+static StringInfo tablespace_map = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
*
- * Essentially what this does is to create a backup label file in $PGDATA,
- * where it will be archived as part of the backup dump. The label file
- * contains the user-supplied label string (typically this would be used
- * to tell where the backup dump will be stored) and the starting time and
- * starting WAL location for the dump.
+ * Essentially what this does is to create the contents required for the
+ * backup_label file and the tablespace map.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -62,7 +60,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +71,40 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state and tablespace_map need to be long-lived as they are used
+ * in pg_backup_stop().
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+
+ /* Allocate backup state or reset it, if it comes from a previous run */
+ if (backup_state == NULL)
+ {
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ }
+ else
+ {
+ MemSet(backup_state, 0, sizeof(BackupState));
+ MemSet(backup_state->name, '\0', sizeof(backup_state->name));
+ }
+
+ /*
+ * tablespace_map may have been created in a previous backup, so take
+ * this occasion to clean it.
+ */
+ if (tablespace_map != NULL)
+ {
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ }
+
+ tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -98,6 +115,15 @@ pg_backup_start(PG_FUNCTION_ARGS)
* allows the user to choose if they want to wait for the WAL to be archived
* or if we should just return as soon as the WAL record is written.
*
+ * This function stops an in-progress backup, creates backup_label contents and
+ * it returns the backup stop LSN, backup_label and tablespace_map contents.
+ *
+ * The backup_label contains the user-supplied label string (typically this
+ * would be used to tell where the backup dump will be stored), the starting
+ * time, starting WAL location for the dump and so on. It is the caller's
+ * responsibility to write the backup_label and tablespace_map files in the
+ * data folder that will be restored from this backup.
+ *
* Permission checking for this function is managed through the normal
* GRANT system.
*/
@@ -108,9 +134,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
+ StringInfo backup_label;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -123,23 +148,28 @@ pg_backup_stop(PG_FUNCTION_ARGS)
errmsg("backup is not in progress"),
errhint("Did you call pg_backup_start()?")));
- /*
- * Stop the backup. Return a copy of the backup label and tablespace map
- * so they can be written to disk by the caller.
- */
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
+ Assert(backup_state != NULL);
+ Assert(tablespace_map != NULL);
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
+ /* Stop the backup */
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /* Build the contents of backup_label */
+ backup_label = build_backup_content(backup_state, false);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_label->data);
+ values[2] = CStringGetTextDatum(tablespace_map->data);
/* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ pfree(backup_state);
+ backup_state = NULL;
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ pfree(backup_label->data);
+ pfree(backup_label);
+ backup_label = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index dd103a8689..703ee08aef 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
+ StringInfo tablespace_map;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +249,21 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ /* Allocate backup related varilables. */
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ tablespace_map = makeStringInfo();
+
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(opt->label, opt->fastcheckpoint, &state.tablespaces,
+ backup_state, tablespace_map);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -313,18 +317,22 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
{
struct stat statbuf;
bool sendtblspclinks = true;
+ StringInfo backup_label;
bbsink_begin_archive(sink, "base.tar");
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
- &manifest);
+ backup_label = build_backup_content(backup_state, false);
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_label->data, &manifest);
+ pfree(backup_label->data);
+ pfree(backup_label);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
- &manifest);
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ tablespace_map->data, &manifest);
sendtblspclinks = false;
}
@@ -374,7 +382,17 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ /* Deallocate backup related variables. */
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ pfree(backup_state);
+ backup_state = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
--
2.37.2
On 2022/09/22 16:43, Michael Paquier wrote:
Added that part before pg_backup_stop() now where it makes sense with
the refactoring.I have put my hands on 0001, and finished with the attached, that
includes many fixes and tweaks. Some of the variable renames felt out
of place, while some comments were overly verbose for their purpose,
though for the last part we did not lose any information in the last
version proposed.
Thanks for updating the patch! This looks better to me.
+ MemSet(backup_state, 0, sizeof(BackupState));
+ MemSet(backup_state->name, '\0', sizeof(backup_state->name));
The latter MemSet() is not necessary because the former already
resets that with zero, is it?
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ }
+
+ tablespace_map = makeStringInfo();
tablespace_map doesn't need to be reset to NULL here.
/* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
<snip>
+ pfree(backup_label->data);
+ pfree(backup_label);
+ backup_label = NULL;
This source comment is a bit misleading, isn't it? Because the memory
for backup_label is allocated under the memory context other than
TopMemoryContext.
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
+#include "utils/memutils.h"
Seems "utils/memutils.h" doesn't need to be included.
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size);
+ appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);
state->stoppoint and state->stoptli should be used instead of
state->startpoint and state->starttli?
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ pfree(backup_state);
+ backup_state = NULL;
It's harmless to set tablespace_map and backup_state to NULL after pfree(),
but it's also unnecessary at least here.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Hi,
On 2022-09-22 16:43:19 +0900, Michael Paquier wrote:
I have put my hands on 0001, and finished with the attached, that
includes many fixes and tweaks.
Due to the merge of the meson based build this patch needs some
adjustment. See
https://cirrus-ci.com/build/6146162607521792
Looks like it just requires adding xlogbackup.c to
src/backend/access/transam/meson.build.
Greetings,
Andres Freund
On Thu, Sep 22, 2022 at 8:55 PM Andres Freund <andres@anarazel.de> wrote:
Due to the merge of the meson based build this patch needs some
adjustment. See
https://cirrus-ci.com/build/6146162607521792
Looks like it just requires adding xlogbackup.c to
src/backend/access/transam/meson.build.
Thanks! I will post a new patch with that change.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
On Thu, Sep 22, 2022 at 3:17 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
+ MemSet(backup_state, 0, sizeof(BackupState)); + MemSet(backup_state->name, '\0', sizeof(backup_state->name));The latter MemSet() is not necessary because the former already
resets that with zero, is it?
Yes, earlier, the name was a palloc'd string but now it is a fixed
array. Removed.
+ pfree(tablespace_map); + tablespace_map = NULL; + } + + tablespace_map = makeStringInfo();tablespace_map doesn't need to be reset to NULL here.
+ pfree(tablespace_map); + tablespace_map = NULL; + pfree(backup_state); + backup_state = NULL;It's harmless to set tablespace_map and backup_state to NULL after pfree(),
but it's also unnecessary at least here.
Yes. But we can retain it for the sake of consistency with the other
places and avoid dangling pointers, if at all any new code gets added
in between it will be useful.
/* Free structures allocated in TopMemoryContext */ - pfree(label_file->data); - pfree(label_file); <snip> + pfree(backup_label->data); + pfree(backup_label); + backup_label = NULL;This source comment is a bit misleading, isn't it? Because the memory
for backup_label is allocated under the memory context other than
TopMemoryContext.
Yes, we can just say /* Deallocate backup-related variables. */. The
pg_backup_start() has the info about the variables being allocated in
TopMemoryContext.
+#include "access/xlog.h" +#include "access/xlog_internal.h" +#include "access/xlogbackup.h" +#include "utils/memutils.h"Seems "utils/memutils.h" doesn't need to be included.
Yes, removed now.
+ XLByteToSeg(state->startpoint, stopsegno, wal_segment_size); + XLogFileName(stopxlogfile, state->starttli, stopsegno, wal_segment_size); + appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n", + LSN_FORMAT_ARGS(state->startpoint), stopxlogfile);state->stoppoint and state->stoptli should be used instead of
state->startpoint and state->starttli?
Nice catch! Corrected.
PSA v12 patch with the above review comments addressed.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v12-0001-Refactor-backup-related-code.patchapplication/x-patch; name=v12-0001-Refactor-backup-related-code.patchDownload
From 3caa5f392e50bdf3f60b4af9710da0004f11ff39 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Thu, 22 Sep 2022 23:44:16 +0000
Subject: [PATCH v12] Refactor backup related code
Refactor backup related code, advantages of doing so are following:
1) backup state is more structured now - all in a single structure,
callers can create backup_label contents whenever required, either
during the pg_backup_start or the pg_backup_stop or in between.
2) no parsing of backup_label file lines now, no error checking
for invalid parsing.
3) backup_label and history file contents have most of the things
in common, they can now be created within a single function.
4) makes backup related code extensible and readable.
This introduces new source files xlogbackup.c/.h for backup related
code and adds the new code in there. The xlog.c file has already grown
to ~9000 LOC (as of this time). Eventually, we would want to move
all the backup related code from xlog.c, xlogfuncs.c, elsewhere to
here.
Author: Bharath Rupireddy
Reviewed-by: Michael Paquier
Reviewed-by: Fujii Masao
Discussion: https://www.postgresql.org/message-id/CALj2ACVqNUEXkaMKyHHOdvScfN9E4LuCWsX_R-YRNfzQ727CdA%40mail.gmail.com
---
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/meson.build | 1 +
src/backend/access/transam/xlog.c | 224 ++++++++----------------
src/backend/access/transam/xlogbackup.c | 80 +++++++++
src/backend/access/transam/xlogfuncs.c | 97 ++++++----
src/backend/backup/basebackup.c | 44 +++--
src/include/access/xlog.h | 10 +-
src/include/access/xlogbackup.h | 42 +++++
8 files changed, 296 insertions(+), 203 deletions(-)
create mode 100644 src/backend/access/transam/xlogbackup.c
create mode 100644 src/include/access/xlogbackup.h
diff --git a/src/backend/access/transam/Makefile b/src/backend/access/transam/Makefile
index 3e5444a6f7..661c55a9db 100644
--- a/src/backend/access/transam/Makefile
+++ b/src/backend/access/transam/Makefile
@@ -29,6 +29,7 @@ OBJS = \
xact.o \
xlog.o \
xlogarchive.o \
+ xlogbackup.o \
xlogfuncs.o \
xloginsert.o \
xlogprefetcher.o \
diff --git a/src/backend/access/transam/meson.build b/src/backend/access/transam/meson.build
index c32169bd2c..63d17b85ee 100644
--- a/src/backend/access/transam/meson.build
+++ b/src/backend/access/transam/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'xact.c',
'xlog.c',
'xlogarchive.c',
+ 'xlogbackup.c',
'xlogfuncs.c',
'xloginsert.c',
'xlogprefetcher.c',
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f32b2124e6..e061554363 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8242,24 +8242,23 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
/*
* do_pg_backup_start is the workhorse of the user-visible pg_backup_start()
* function. It creates the necessary starting checkpoint and constructs the
- * backup label and tablespace map.
+ * backup state and tablespace map.
*
- * Input parameters are "backupidstr" (the backup label string) and "fast"
- * (if true, we do the checkpoint in immediate mode to make it faster).
+ * Input parameters are "state" (the backup state), "fast" (if true, we do
+ * the checkpoint in immediate mode to make it faster), and "tablespaces"
+ * (if non-NULL, indicates a list of tablespaceinfo structs describing the
+ * cluster's tablespaces.).
*
- * The backup label and tablespace map contents are appended to *labelfile and
- * *tblspcmapfile, and the caller is responsible for including them in the
- * backup archive as 'backup_label' and 'tablespace_map'.
- * tblspcmapfile is required mainly for tar format in windows as native windows
- * utilities are not able to create symlinks while extracting files from tar.
- * However for consistency and platform-independence, we do it the same way
- * everywhere.
+ * The tablespace map contents are appended to passed-in parameter
+ * tablespace_map and the caller is responsible for including it in the backup
+ * archive as 'tablespace_map'. The tablespace_map file is required mainly for
+ * tar format in windows as native windows utilities are not able to create
+ * symlinks while extracting files from tar. However for consistency and
+ * platform-independence, we do it the same way everywhere.
*
- * If "tablespaces" isn't NULL, it receives a list of tablespaceinfo structs
- * describing the cluster's tablespaces.
- *
- * Returns the minimum WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *starttli_p.
+ * It fills in backup_state with the information required for the backup,
+ * such as the minimum WAL location that must be present to restore from
+ * this backup (starttli) and the corresponding timeline ID (starttli).
*
* Every successfully started backup must be stopped by calling
* do_pg_backup_stop() or do_pg_abort_backup(). There can be many
@@ -8268,20 +8267,13 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
- StringInfo labelfile, List **tablespaces,
- StringInfo tblspcmapfile)
+void
+do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
+ BackupState *state, StringInfo tblspcmapfile)
{
bool backup_started_in_recovery = false;
- XLogRecPtr checkpointloc;
- XLogRecPtr startpoint;
- TimeLineID starttli;
- pg_time_t stamp_time;
- char strfbuf[128];
- char xlogfilename[MAXFNAMELEN];
- XLogSegNo _logSegNo;
+ Assert(state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8300,6 +8292,8 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
errmsg("backup label too long (max %d bytes)",
MAXPGPATH)));
+ memcpy(state->name, backupidstr, strlen(backupidstr));
+
/*
* Mark backup active in shared memory. We must do full-page WAL writes
* during an on-line backup even if not doing so at other times, because
@@ -8391,9 +8385,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- checkpointloc = ControlFile->checkPoint;
- startpoint = ControlFile->checkPointCopy.redo;
- starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ state->checkpointloc = ControlFile->checkPoint;
+ state->startpoint = ControlFile->checkPointCopy.redo;
+ state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8410,7 +8404,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || startpoint <= recptr)
+ if (!checkpointfpw || state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8442,17 +8436,14 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < startpoint)
+ if (XLogCtl->Insert.lastBackupStart < state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = startpoint;
+ XLogCtl->Insert.lastBackupStart = state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
} while (!gotUniqueStartpoint);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- XLogFileName(xlogfilename, starttli, _logSegNo, wal_segment_size);
-
/*
* Construct tablespace_map file.
*/
@@ -8538,39 +8529,16 @@ do_pg_backup_start(const char *backupidstr, bool fast, TimeLineID *starttli_p,
}
FreeDir(tblspcdir);
- /*
- * Construct backup label file.
- */
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
- appendStringInfo(labelfile, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), xlogfilename);
- appendStringInfo(labelfile, "CHECKPOINT LOCATION: %X/%X\n",
- LSN_FORMAT_ARGS(checkpointloc));
- appendStringInfo(labelfile, "BACKUP METHOD: streamed\n");
- appendStringInfo(labelfile, "BACKUP FROM: %s\n",
- backup_started_in_recovery ? "standby" : "primary");
- appendStringInfo(labelfile, "START TIME: %s\n", strfbuf);
- appendStringInfo(labelfile, "LABEL: %s\n", backupidstr);
- appendStringInfo(labelfile, "START TIMELINE: %u\n", starttli);
+ state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
+ state->started_in_recovery = backup_started_in_recovery;
+
/*
* Mark that the start phase has correctly finished for the backup.
*/
sessionBackupState = SESSION_BACKUP_RUNNING;
-
- /*
- * We're done. As a convenience, return the starting WAL location.
- */
- if (starttli_p)
- *starttli_p = starttli;
- return startpoint;
}
/* Error cleanup callback for pg_backup_start */
@@ -8602,48 +8570,38 @@ get_backup_status(void)
/*
* do_pg_backup_stop
*
- * Utility function called at the end of an online backup. It cleans up the
- * backup state and can optionally wait for WAL segments to be archived.
+ * Utility function called at the end of an online backup. It creates history
+ * file (if required), resets sessionBackupState and so on. It can optionally
+ * wait for WAL segments to be archived.
*
- * Returns the last WAL location that must be present to restore from this
- * backup, and the corresponding timeline ID in *stoptli_p.
+ * backup_state is filled with the information necessary to restore from this
+ * backup with its stop LSN (stoppoint), its timeline ID (stoptli), etc.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
*/
-XLogRecPtr
-do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
+void
+do_pg_backup_stop(BackupState *state, bool waitforarchive)
{
- bool backup_started_in_recovery = false;
- XLogRecPtr startpoint;
- XLogRecPtr stoppoint;
- TimeLineID stoptli;
- pg_time_t stamp_time;
- char strfbuf[128];
+ bool backup_stopped_in_recovery = false;
char histfilepath[MAXPGPATH];
- char startxlogfilename[MAXFNAMELEN];
- char stopxlogfilename[MAXFNAMELEN];
char lastxlogfilename[MAXFNAMELEN];
char histfilename[MAXFNAMELEN];
- char backupfrom[20];
XLogSegNo _logSegNo;
FILE *fp;
- char ch;
int seconds_before_warning;
int waits = 0;
bool reported_waiting = false;
- char *remaining;
- char *ptr;
- uint32 hi,
- lo;
- backup_started_in_recovery = RecoveryInProgress();
+ Assert(state != NULL);
+
+ backup_stopped_in_recovery = RecoveryInProgress();
/*
* During recovery, we don't need to check WAL level. Because, if WAL
* level is not sufficient, it's impossible to get here during recovery.
*/
- if (!backup_started_in_recovery && !XLogIsNeeded())
+ if (!backup_stopped_in_recovery && !XLogIsNeeded())
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL level not sufficient for making an online backup"),
@@ -8684,29 +8642,11 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
WALInsertLockRelease();
/*
- * Read and parse the START WAL LOCATION line (this code is pretty crude,
- * but we are not expecting any variability in the file format).
+ * If we are taking an online backup from the standby, we confirm that the
+ * standby has not been promoted during the backup.
*/
- if (sscanf(labelfile, "START WAL LOCATION: %X/%X (file %24s)%c",
- &hi, &lo, startxlogfilename,
- &ch) != 4 || ch != '\n')
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- startpoint = ((uint64) hi) << 32 | lo;
- remaining = strchr(labelfile, '\n') + 1; /* %n is not portable enough */
-
- /*
- * Parse the BACKUP FROM line. If we are taking an online backup from the
- * standby, we confirm that the standby has not been promoted during the
- * backup.
- */
- ptr = strstr(remaining, "BACKUP FROM:");
- if (!ptr || sscanf(ptr, "BACKUP FROM: %19s\n", backupfrom) != 1)
- ereport(ERROR,
- (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("invalid data in file \"%s\"", BACKUP_LABEL_FILE)));
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("the standby was promoted during online backup"),
@@ -8742,7 +8682,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
* an archiver is not invoked. So it doesn't seem worthwhile to write a
* backup history file during recovery.
*/
- if (backup_started_in_recovery)
+ if (backup_stopped_in_recovery)
{
XLogRecPtr recptr;
@@ -8754,7 +8694,7 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (startpoint <= recptr)
+ if (state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8766,24 +8706,27 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
LWLockAcquire(ControlFileLock, LW_SHARED);
- stoppoint = ControlFile->minRecoveryPoint;
- stoptli = ControlFile->minRecoveryPointTLI;
+ state->stoppoint = ControlFile->minRecoveryPoint;
+ state->stoptli = ControlFile->minRecoveryPointTLI;
LWLockRelease(ControlFileLock);
}
else
{
+ StringInfo history_file;
+
/*
* Write the backup-end xlog record
*/
XLogBeginInsert();
- XLogRegisterData((char *) (&startpoint), sizeof(startpoint));
- stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
+ XLogRegisterData((char *) (&state->startpoint),
+ sizeof(state->startpoint));
+ state->stoppoint = XLogInsert(RM_XLOG_ID, XLOG_BACKUP_END);
/*
* Given that we're not in recovery, InsertTimeLineID is set and can't
* change, so we can read it without a lock.
*/
- stoptli = XLogCtl->InsertTimeLineID;
+ state->stoptli = XLogCtl->InsertTimeLineID;
/*
* Force a switch to a new xlog segment file, so that the backup is
@@ -8791,39 +8734,28 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
RequestXLogSwitch(false);
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(stopxlogfilename, stoptli, _logSegNo, wal_segment_size);
-
- /* Use the log timezone here, not the session timezone */
- stamp_time = (pg_time_t) time(NULL);
- pg_strftime(strfbuf, sizeof(strfbuf),
- "%Y-%m-%d %H:%M:%S %Z",
- pg_localtime(&stamp_time, log_timezone));
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ state->stoptime = (pg_time_t) time(NULL);
/*
- * Write the backup history file
+ * Write the backup history file. Add backup_label file contents too it.
*/
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFilePath(histfilepath, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFilePath(histfilepath, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
fp = AllocateFile(histfilepath, "w");
if (!fp)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not create file \"%s\": %m",
histfilepath)));
- fprintf(fp, "START WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(startpoint), startxlogfilename);
- fprintf(fp, "STOP WAL LOCATION: %X/%X (file %s)\n",
- LSN_FORMAT_ARGS(stoppoint), stopxlogfilename);
- /*
- * Transfer remaining lines including label and start timeline to
- * history file.
- */
- fprintf(fp, "%s", remaining);
- fprintf(fp, "STOP TIME: %s\n", strfbuf);
- fprintf(fp, "STOP TIMELINE: %u\n", stoptli);
+ /* Build and save the contents of the backup history file */
+ history_file = build_backup_content(state, true);
+ fprintf(fp, "%s", history_file->data);
+ pfree(history_file->data);
+ pfree(history_file);
+
if (fflush(fp) || ferror(fp) || FreeFile(fp))
ereport(ERROR,
(errcode_for_file_access(),
@@ -8861,15 +8793,16 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
*/
if (waitforarchive &&
- ((!backup_started_in_recovery && XLogArchivingActive()) ||
- (backup_started_in_recovery && XLogArchivingAlways())))
+ ((!backup_stopped_in_recovery && XLogArchivingActive()) ||
+ (backup_stopped_in_recovery && XLogArchivingAlways())))
{
- XLByteToPrevSeg(stoppoint, _logSegNo, wal_segment_size);
- XLogFileName(lastxlogfilename, stoptli, _logSegNo, wal_segment_size);
+ XLByteToPrevSeg(state->stoppoint, _logSegNo, wal_segment_size);
+ XLogFileName(lastxlogfilename, state->stoptli, _logSegNo,
+ wal_segment_size);
- XLByteToSeg(startpoint, _logSegNo, wal_segment_size);
- BackupHistoryFileName(histfilename, stoptli, _logSegNo,
- startpoint, wal_segment_size);
+ XLByteToSeg(state->startpoint, _logSegNo, wal_segment_size);
+ BackupHistoryFileName(histfilename, state->stoptli, _logSegNo,
+ state->startpoint, wal_segment_size);
seconds_before_warning = 60;
waits = 0;
@@ -8910,13 +8843,6 @@ do_pg_backup_stop(char *labelfile, bool waitforarchive, TimeLineID *stoptli_p)
else if (waitforarchive)
ereport(NOTICE,
(errmsg("WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup")));
-
- /*
- * We're done. As a convenience, return the ending WAL location.
- */
- if (stoptli_p)
- *stoptli_p = stoptli;
- return stoppoint;
}
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
new file mode 100644
index 0000000000..832aacab0b
--- /dev/null
+++ b/src/backend/access/transam/xlogbackup.c
@@ -0,0 +1,80 @@
+/*-------------------------------------------------------------------------
+ *
+ * xlogbackup.c
+ * Internal routines for base backups.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/transam/xlogbackup.c
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/xlog.h"
+#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
+
+/*
+ * Build contents for backup_label or backup history file.
+ *
+ * When ishistoryfile is true, it creates the contents for a backup history
+ * file, otherwise it creates contents for a backup_label file.
+ *
+ * Returns the result generated as a palloc'd StringInfo.
+ */
+StringInfo
+build_backup_content(BackupState *state, bool ishistoryfile)
+{
+ char startstrbuf[128];
+ char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
+ XLogSegNo startsegno;
+ StringInfo result = makeStringInfo();
+
+ Assert(state != NULL);
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(startstrbuf, sizeof(startstrbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->starttime, log_timezone));
+
+ XLByteToSeg(state->startpoint, startsegno, wal_segment_size);
+ XLogFileName(startxlogfile, state->starttli, startsegno, wal_segment_size);
+ appendStringInfo(result, "START WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->startpoint), startxlogfile);
+
+ if (ishistoryfile)
+ {
+ char stopxlogfile[MAXFNAMELEN]; /* backup stop WAL file */
+ XLogSegNo stopsegno;
+
+ XLByteToSeg(state->stoppoint, stopsegno, wal_segment_size);
+ XLogFileName(stopxlogfile, state->stoptli, stopsegno, wal_segment_size);
+ appendStringInfo(result, "STOP WAL LOCATION: %X/%X (file %s)\n",
+ LSN_FORMAT_ARGS(state->stoppoint), stopxlogfile);
+ }
+
+ appendStringInfo(result, "CHECKPOINT LOCATION: %X/%X\n",
+ LSN_FORMAT_ARGS(state->checkpointloc));
+ appendStringInfo(result, "BACKUP METHOD: streamed\n");
+ appendStringInfo(result, "BACKUP FROM: %s\n",
+ state->started_in_recovery ? "standby" : "primary");
+ appendStringInfo(result, "START TIME: %s\n", startstrbuf);
+ appendStringInfo(result, "LABEL: %s\n", state->name);
+ appendStringInfo(result, "START TIMELINE: %u\n", state->starttli);
+
+ if (ishistoryfile)
+ {
+ char stopstrfbuf[128];
+
+ /* Use the log timezone here, not the session timezone */
+ pg_strftime(stopstrfbuf, sizeof(stopstrfbuf), "%Y-%m-%d %H:%M:%S %Z",
+ pg_localtime(&state->stoptime, log_timezone));
+
+ appendStringInfo(result, "STOP TIME: %s\n", stopstrfbuf);
+ appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli);
+ }
+
+ return result;
+}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index 27aeb6e281..be29812d9e 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -20,6 +20,7 @@
#include "access/htup_details.h"
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "access/xlogrecovery.h"
#include "access/xlogutils.h"
#include "catalog/pg_type.h"
@@ -39,19 +40,16 @@
#include "utils/tuplestore.h"
/*
- * Store label file and tablespace map during backups.
+ * Backup-related variables.
*/
-static StringInfo label_file;
-static StringInfo tblspc_map_file;
+static BackupState *backup_state = NULL;
+static StringInfo tablespace_map = NULL;
/*
* pg_backup_start: set up for taking an on-line backup dump
*
- * Essentially what this does is to create a backup label file in $PGDATA,
- * where it will be archived as part of the backup dump. The label file
- * contains the user-supplied label string (typically this would be used
- * to tell where the backup dump will be stored) and the starting time and
- * starting WAL location for the dump.
+ * Essentially what this does is to create the contents required for the
+ * backup_label file and the tablespace map.
*
* Permission checking for this function is managed through the normal
* GRANT system.
@@ -62,7 +60,6 @@ pg_backup_start(PG_FUNCTION_ARGS)
text *backupid = PG_GETARG_TEXT_PP(0);
bool fast = PG_GETARG_BOOL(1);
char *backupidstr;
- XLogRecPtr startpoint;
SessionBackupState status = get_backup_status();
MemoryContext oldcontext;
@@ -74,20 +71,35 @@ pg_backup_start(PG_FUNCTION_ARGS)
errmsg("a backup is already in progress in this session")));
/*
- * Label file and tablespace map file need to be long-lived, since they
- * are read in pg_backup_stop.
+ * backup_state and tablespace_map need to be long-lived as they are used
+ * in pg_backup_stop().
*/
oldcontext = MemoryContextSwitchTo(TopMemoryContext);
- label_file = makeStringInfo();
- tblspc_map_file = makeStringInfo();
+
+ /* Allocate backup state or reset it, if it comes from a previous run */
+ if (backup_state == NULL)
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ else
+ MemSet(backup_state, 0, sizeof(BackupState));
+
+ /*
+ * tablespace_map may have been created in a previous backup, so take
+ * this occasion to clean it.
+ */
+ if (tablespace_map != NULL)
+ {
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ }
+
+ tablespace_map = makeStringInfo();
MemoryContextSwitchTo(oldcontext);
register_persistent_abort_backup_handler();
+ do_pg_backup_start(backupidstr, fast, NULL, backup_state, tablespace_map);
- startpoint = do_pg_backup_start(backupidstr, fast, NULL, label_file,
- NULL, tblspc_map_file);
-
- PG_RETURN_LSN(startpoint);
+ PG_RETURN_LSN(backup_state->startpoint);
}
@@ -98,6 +110,15 @@ pg_backup_start(PG_FUNCTION_ARGS)
* allows the user to choose if they want to wait for the WAL to be archived
* or if we should just return as soon as the WAL record is written.
*
+ * This function stops an in-progress backup, creates backup_label contents and
+ * it returns the backup stop LSN, backup_label and tablespace_map contents.
+ *
+ * The backup_label contains the user-supplied label string (typically this
+ * would be used to tell where the backup dump will be stored), the starting
+ * time, starting WAL location for the dump and so on. It is the caller's
+ * responsibility to write the backup_label and tablespace_map files in the
+ * data folder that will be restored from this backup.
+ *
* Permission checking for this function is managed through the normal
* GRANT system.
*/
@@ -108,9 +129,8 @@ pg_backup_stop(PG_FUNCTION_ARGS)
TupleDesc tupdesc;
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
-
bool waitforarchive = PG_GETARG_BOOL(0);
- XLogRecPtr stoppoint;
+ StringInfo backup_label;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -123,23 +143,28 @@ pg_backup_stop(PG_FUNCTION_ARGS)
errmsg("backup is not in progress"),
errhint("Did you call pg_backup_start()?")));
- /*
- * Stop the backup. Return a copy of the backup label and tablespace map
- * so they can be written to disk by the caller.
- */
- stoppoint = do_pg_backup_stop(label_file->data, waitforarchive, NULL);
-
- values[0] = LSNGetDatum(stoppoint);
- values[1] = CStringGetTextDatum(label_file->data);
- values[2] = CStringGetTextDatum(tblspc_map_file->data);
-
- /* Free structures allocated in TopMemoryContext */
- pfree(label_file->data);
- pfree(label_file);
- label_file = NULL;
- pfree(tblspc_map_file->data);
- pfree(tblspc_map_file);
- tblspc_map_file = NULL;
+ Assert(backup_state != NULL);
+ Assert(tablespace_map != NULL);
+
+ /* Stop the backup */
+ do_pg_backup_stop(backup_state, waitforarchive);
+
+ /* Build the contents of backup_label */
+ backup_label = build_backup_content(backup_state, false);
+
+ values[0] = LSNGetDatum(backup_state->stoppoint);
+ values[1] = CStringGetTextDatum(backup_label->data);
+ values[2] = CStringGetTextDatum(tablespace_map->data);
+
+ /* Deallocate backup-related variables. */
+ pfree(backup_state);
+ backup_state = NULL;
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ pfree(backup_label->data);
+ pfree(backup_label);
+ backup_label = NULL;
/* Returns the record as Datum */
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index dd103a8689..5b72b5f8b4 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -17,6 +17,7 @@
#include <time.h>
#include "access/xlog_internal.h"
+#include "access/xlogbackup.h"
#include "backup/backup_manifest.h"
#include "backup/basebackup.h"
#include "backup/basebackup_sink.h"
@@ -231,9 +232,9 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
bbsink_state state;
XLogRecPtr endptr;
TimeLineID endtli;
- StringInfo labelfile;
- StringInfo tblspc_map_file;
backup_manifest_info manifest;
+ BackupState *backup_state;
+ StringInfo tablespace_map;
/* Initial backup state, insofar as we know it now. */
state.tablespaces = NIL;
@@ -248,18 +249,21 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
backup_started_in_recovery = RecoveryInProgress();
- labelfile = makeStringInfo();
- tblspc_map_file = makeStringInfo();
InitializeBackupManifest(&manifest, opt->manifest,
opt->manifest_checksum_type);
total_checksum_failures = 0;
+ /* Allocate backup related varilables. */
+ backup_state = (BackupState *) palloc0(sizeof(BackupState));
+ tablespace_map = makeStringInfo();
+
basebackup_progress_wait_checkpoint();
- state.startptr = do_pg_backup_start(opt->label, opt->fastcheckpoint,
- &state.starttli,
- labelfile, &state.tablespaces,
- tblspc_map_file);
+ do_pg_backup_start(opt->label, opt->fastcheckpoint, &state.tablespaces,
+ backup_state, tablespace_map);
+
+ state.startptr = backup_state->startpoint;
+ state.starttli = backup_state->starttli;
/*
* Once do_pg_backup_start has been called, ensure that any failure causes
@@ -313,18 +317,22 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
{
struct stat statbuf;
bool sendtblspclinks = true;
+ StringInfo backup_label;
bbsink_begin_archive(sink, "base.tar");
/* In the main tar, include the backup_label first... */
- sendFileWithContent(sink, BACKUP_LABEL_FILE, labelfile->data,
- &manifest);
+ backup_label = build_backup_content(backup_state, false);
+ sendFileWithContent(sink, BACKUP_LABEL_FILE,
+ backup_label->data, &manifest);
+ pfree(backup_label->data);
+ pfree(backup_label);
/* Then the tablespace_map file, if required... */
if (opt->sendtblspcmapfile)
{
- sendFileWithContent(sink, TABLESPACE_MAP, tblspc_map_file->data,
- &manifest);
+ sendFileWithContent(sink, TABLESPACE_MAP,
+ tablespace_map->data, &manifest);
sendtblspclinks = false;
}
@@ -374,7 +382,17 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
}
basebackup_progress_wait_wal_archive(&state);
- endptr = do_pg_backup_stop(labelfile->data, !opt->nowait, &endtli);
+ do_pg_backup_stop(backup_state, !opt->nowait);
+
+ endptr = backup_state->stoppoint;
+ endtli = backup_state->stoptli;
+
+ /* Deallocate backup-related variables. */
+ pfree(tablespace_map->data);
+ pfree(tablespace_map);
+ tablespace_map = NULL;
+ pfree(backup_state);
+ backup_state = NULL;
}
PG_END_ENSURE_ERROR_CLEANUP(do_pg_abort_backup, BoolGetDatum(false));
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3dbfa6b593..dce265098e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -11,6 +11,7 @@
#ifndef XLOG_H
#define XLOG_H
+#include "access/xlogbackup.h"
#include "access/xlogdefs.h"
#include "access/xlogreader.h"
#include "datatype/timestamp.h"
@@ -277,11 +278,10 @@ typedef enum SessionBackupState
SESSION_BACKUP_RUNNING,
} SessionBackupState;
-extern XLogRecPtr do_pg_backup_start(const char *backupidstr, bool fast,
- TimeLineID *starttli_p, StringInfo labelfile,
- List **tablespaces, StringInfo tblspcmapfile);
-extern XLogRecPtr do_pg_backup_stop(char *labelfile, bool waitforarchive,
- TimeLineID *stoptli_p);
+extern void do_pg_backup_start(const char *backupidstr, bool fast,
+ List **tablespaces, BackupState *state,
+ StringInfo tblspcmapfile);
+extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
extern void register_persistent_abort_backup_handler(void);
extern SessionBackupState get_backup_status(void);
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
new file mode 100644
index 0000000000..ccdfefe117
--- /dev/null
+++ b/src/include/access/xlogbackup.h
@@ -0,0 +1,42 @@
+/*-------------------------------------------------------------------------
+ *
+ * xlogbackup.h
+ * Definitions for internals of base backups.
+ *
+ * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/include/access/xlogbackup.h
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef XLOG_BACKUP_H
+#define XLOG_BACKUP_H
+
+#include "access/xlogdefs.h"
+#include "lib/stringinfo.h"
+#include "pgtime.h"
+
+/* Structure to hold backup state. */
+typedef struct BackupState
+{
+ /* Fields saved at backup start */
+ /* Backup label name one extra byte for null-termination */
+ char name[MAXPGPATH + 1];
+ XLogRecPtr startpoint; /* backup start WAL location */
+ TimeLineID starttli; /* backup start TLI */
+ XLogRecPtr checkpointloc; /* last checkpoint location */
+ pg_time_t starttime; /* backup start time */
+ bool started_in_recovery; /* backup started in recovery? */
+
+ /* Fields saved at the end of backup */
+ XLogRecPtr stoppoint; /* backup stop WAL location */
+ TimeLineID stoptli; /* backup stop TLI */
+ pg_time_t stoptime; /* backup stop time */
+} BackupState;
+
+extern StringInfo build_backup_content(BackupState *state,
+ bool ishistoryfile);
+
+#endif /* XLOG_BACKUP_H */
--
2.34.1
On Fri, Sep 23, 2022 at 06:02:24AM +0530, Bharath Rupireddy wrote:
PSA v12 patch with the above review comments addressed.
I've read this one, and nothing is standing out at quick glance, so
that looks rather reasonable to me. I should be able to spend more
time on that at the beginning of next week, and maybe apply it.
--
Michael
On Thu, Sep 22, 2022 at 08:25:31AM -0700, Andres Freund wrote:
Due to the merge of the meson based build this patch needs some
adjustment. See
https://cirrus-ci.com/build/6146162607521792
Looks like it just requires adding xlogbackup.c to
src/backend/access/transam/meson.build.
Thanks for the reminder. I have played a bit with meson and ninja,
and that's a rather straight-forward experience. The output is muuuch
nicer to parse.
--
Michael
On Fri, Sep 23, 2022 at 09:12:22PM +0900, Michael Paquier wrote:
I've read this one, and nothing is standing out at quick glance, so
that looks rather reasonable to me. I should be able to spend more
time on that at the beginning of next week, and maybe apply it.
What I had at hand seemed fine on a second look, so applied after
tweaking a couple of comments. One thing that I have been wondering
after-the-fact is whether it would be cleaner to return the contents
of the backup history file or backup_label as a char rather than a
StringInfo? This simplifies a bit what the callers of
build_backup_content() need to do.
--
Michael
Attachments:
xlogbackup-simpler.patchtext/x-diff; charset=us-asciiDownload
diff --git a/src/include/access/xlogbackup.h b/src/include/access/xlogbackup.h
index cb15b8b80a..8ec3d88b0a 100644
--- a/src/include/access/xlogbackup.h
+++ b/src/include/access/xlogbackup.h
@@ -15,7 +15,6 @@
#define XLOG_BACKUP_H
#include "access/xlogdefs.h"
-#include "lib/stringinfo.h"
#include "pgtime.h"
/* Structure to hold backup state. */
@@ -36,7 +35,7 @@ typedef struct BackupState
pg_time_t stoptime; /* backup stop time */
} BackupState;
-extern StringInfo build_backup_content(BackupState *state,
- bool ishistoryfile);
+extern char *build_backup_content(BackupState *state,
+ bool ishistoryfile);
#endif /* XLOG_BACKUP_H */
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 7606ee128a..1dd6df0fe1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8711,7 +8711,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive)
}
else
{
- StringInfo history_file;
+ char *history_file;
/*
* Write the backup-end xlog record
@@ -8751,8 +8751,7 @@ do_pg_backup_stop(BackupState *state, bool waitforarchive)
/* Build and save the contents of the backup history file */
history_file = build_backup_content(state, true);
- fprintf(fp, "%s", history_file->data);
- pfree(history_file->data);
+ fprintf(fp, "%s", history_file);
pfree(history_file);
if (fflush(fp) || ferror(fp) || FreeFile(fp))
diff --git a/src/backend/access/transam/xlogbackup.c b/src/backend/access/transam/xlogbackup.c
index c01c1f9010..90b5273b02 100644
--- a/src/backend/access/transam/xlogbackup.c
+++ b/src/backend/access/transam/xlogbackup.c
@@ -23,15 +23,16 @@
* When ishistoryfile is true, it creates the contents for a backup history
* file, otherwise it creates contents for a backup_label file.
*
- * Returns the result generated as a palloc'd StringInfo.
+ * Returns the result generated as a palloc'd string.
*/
-StringInfo
+char *
build_backup_content(BackupState *state, bool ishistoryfile)
{
char startstrbuf[128];
char startxlogfile[MAXFNAMELEN]; /* backup start WAL file */
XLogSegNo startsegno;
StringInfo result = makeStringInfo();
+ char *data;
Assert(state != NULL);
@@ -76,5 +77,8 @@ build_backup_content(BackupState *state, bool ishistoryfile)
appendStringInfo(result, "STOP TIMELINE: %u\n", state->stoptli);
}
- return result;
+ data = result->data;
+ pfree(result);
+
+ return data;
}
diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c
index f724b18733..a801a94fe8 100644
--- a/src/backend/access/transam/xlogfuncs.c
+++ b/src/backend/access/transam/xlogfuncs.c
@@ -130,7 +130,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
Datum values[PG_BACKUP_STOP_V2_COLS] = {0};
bool nulls[PG_BACKUP_STOP_V2_COLS] = {0};
bool waitforarchive = PG_GETARG_BOOL(0);
- StringInfo backup_label;
+ char *backup_label;
SessionBackupState status = get_backup_status();
/* Initialize attributes information in the tuple descriptor */
@@ -153,7 +153,7 @@ pg_backup_stop(PG_FUNCTION_ARGS)
backup_label = build_backup_content(backup_state, false);
values[0] = LSNGetDatum(backup_state->stoppoint);
- values[1] = CStringGetTextDatum(backup_label->data);
+ values[1] = CStringGetTextDatum(backup_label);
values[2] = CStringGetTextDatum(tablespace_map->data);
/* Deallocate backup-related variables */
@@ -162,7 +162,6 @@ pg_backup_stop(PG_FUNCTION_ARGS)
pfree(tablespace_map->data);
pfree(tablespace_map);
tablespace_map = NULL;
- pfree(backup_label->data);
pfree(backup_label);
/* Returns the record as Datum */
diff --git a/src/backend/backup/basebackup.c b/src/backend/backup/basebackup.c
index 495bbb506a..411cac9be3 100644
--- a/src/backend/backup/basebackup.c
+++ b/src/backend/backup/basebackup.c
@@ -317,15 +317,14 @@ perform_base_backup(basebackup_options *opt, bbsink *sink)
{
struct stat statbuf;
bool sendtblspclinks = true;
- StringInfo backup_label;
+ char *backup_label;
bbsink_begin_archive(sink, "base.tar");
/* In the main tar, include the backup_label first... */
backup_label = build_backup_content(backup_state, false);
sendFileWithContent(sink, BACKUP_LABEL_FILE,
- backup_label->data, &manifest);
- pfree(backup_label->data);
+ backup_label, &manifest);
pfree(backup_label);
/* Then the tablespace_map file, if required... */
On Mon, Sep 26, 2022 at 8:13 AM Michael Paquier <michael@paquier.xyz> wrote:
What I had at hand seemed fine on a second look, so applied after
tweaking a couple of comments. One thing that I have been wondering
after-the-fact is whether it would be cleaner to return the contents
of the backup history file or backup_label as a char rather than a
StringInfo? This simplifies a bit what the callers of
build_backup_content() need to do.
+1 because callers don't use returned StringInfo structure outside of
build_backup_content(). The patch looks good to me. I think it will be
good to add a note about the caller freeing up the retired string's
memory [1]* Returns the result generated as a palloc'd string. It is the caller's * responsibility to free the returned string's memory. */ char * build_backup_content(BackupState *state, bool ishistoryfile) {, just in case.
[1]: * Returns the result generated as a palloc'd string. It is the caller's * responsibility to free the returned string's memory. */ char * build_backup_content(BackupState *state, bool ishistoryfile) {
* Returns the result generated as a palloc'd string. It is the caller's
* responsibility to free the returned string's memory.
*/
char *
build_backup_content(BackupState *state, bool ishistoryfile)
{
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
At Mon, 26 Sep 2022 11:57:58 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
On Mon, Sep 26, 2022 at 8:13 AM Michael Paquier <michael@paquier.xyz> wrote:
What I had at hand seemed fine on a second look, so applied after
tweaking a couple of comments. One thing that I have been wondering
after-the-fact is whether it would be cleaner to return the contents
of the backup history file or backup_label as a char rather than a
StringInfo? This simplifies a bit what the callers of
build_backup_content() need to do.+1 because callers don't use returned StringInfo structure outside of
build_backup_content(). The patch looks good to me. I think it will be
good to add a note about the caller freeing up the retired string's
memory [1], just in case.
Doesn't the following (from you :) work?
+ * Returns the result generated as a palloc'd string.
This suggests no need for pfree if the caller properly destroys the
context or pfree is needed otherwise. In this case, the active memory
contexts are "ExprContext" and "Replication command context" so I
think we actually do not need to pfree it but I don't mean we sholnd't
do that in this patch (since those contexts are somewhat remote from
what the function does and pfree doesn't matter at all here.).
[1]
* Returns the result generated as a palloc'd string. It is the caller's
* responsibility to free the returned string's memory.
*/
char *
build_backup_content(BackupState *state, bool ishistoryfile)
{
+1. A nitpick.
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery)
+ if (state->started_in_recovery == true &&
+ backup_stopped_in_recovery == false)
Using == for Booleans may not be great?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Mon, Sep 26, 2022 at 05:10:16PM +0900, Kyotaro Horiguchi wrote:
- if (strcmp(backupfrom, "standby") == 0 && !backup_started_in_recovery) + if (state->started_in_recovery == true && + backup_stopped_in_recovery == false)Using == for Booleans may not be great?
Yes. That's why 7d70809 does not use the way proposed by the previous
patch.
--
Michael
On Mon, Sep 26, 2022 at 11:57:58AM +0530, Bharath Rupireddy wrote:
+1 because callers don't use returned StringInfo structure outside of
build_backup_content(). The patch looks good to me.
Thanks for looking.
I think it will be
good to add a note about the caller freeing up the retired string's
memory [1], just in case.
Not sure that this is worth it. It is fine to use palloc() in a
dedicated memory context, for one.
--
Michael
This commit introduced BackupState struct. The comment of
do_pg_backup_start says that:
* It fills in backup_state with the information required for the backup,
And the parameters are:
do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
BackupState *state, StringInfo tblspcmapfile)
So backup_state is different from both the type BackupState and the
parameter state. I find it annoying. Don't we either rename the
parameter or fix the comment?
The parameter "state" sounds a bit too generic. So I prefer to rename
the parameter to backup_state, as the attached.
What do you think about this?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
Attachments:
rename_param_of_po_pg_backup_start.txttext/plain; charset=us-asciiDownload
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1dd6df0fe1..715d5868eb 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8244,10 +8244,10 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
* function. It creates the necessary starting checkpoint and constructs the
* backup state and tablespace map.
*
- * Input parameters are "state" (the backup state), "fast" (if true, we do
- * the checkpoint in immediate mode to make it faster), and "tablespaces"
- * (if non-NULL, indicates a list of tablespaceinfo structs describing the
- * cluster's tablespaces.).
+ * Input parameters are "backup_state", "fast" (if true, we do the checkpoint
+ * in immediate mode to make it faster), and "tablespaces" (if non-NULL,
+ * indicates a list of tablespaceinfo structs describing the cluster's
+ * tablespaces.).
*
* The tablespace map contents are appended to passed-in parameter
* tablespace_map and the caller is responsible for including it in the backup
@@ -8269,11 +8269,11 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
*/
void
do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
- BackupState *state, StringInfo tblspcmapfile)
+ BackupState *backup_state, StringInfo tblspcmapfile)
{
bool backup_started_in_recovery = false;
- Assert(state != NULL);
+ Assert(backup_state != NULL);
backup_started_in_recovery = RecoveryInProgress();
/*
@@ -8292,7 +8292,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
errmsg("backup label too long (max %d bytes)",
MAXPGPATH)));
- memcpy(state->name, backupidstr, strlen(backupidstr));
+ memcpy(backup_state->name, backupidstr, strlen(backupidstr));
/*
* Mark backup active in shared memory. We must do full-page WAL writes
@@ -8385,9 +8385,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
* pointer.
*/
LWLockAcquire(ControlFileLock, LW_SHARED);
- state->checkpointloc = ControlFile->checkPoint;
- state->startpoint = ControlFile->checkPointCopy.redo;
- state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
+ backup_state->checkpointloc = ControlFile->checkPoint;
+ backup_state->startpoint = ControlFile->checkPointCopy.redo;
+ backup_state->starttli = ControlFile->checkPointCopy.ThisTimeLineID;
checkpointfpw = ControlFile->checkPointCopy.fullPageWrites;
LWLockRelease(ControlFileLock);
@@ -8404,7 +8404,7 @@ do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
recptr = XLogCtl->lastFpwDisableRecPtr;
SpinLockRelease(&XLogCtl->info_lck);
- if (!checkpointfpw || state->startpoint <= recptr)
+ if (!checkpointfpw || backup_state->startpoint <= recptr)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("WAL generated with full_page_writes=off was replayed "
@@ -8436,9 +8436,9 @@ do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
* either because only few buffers have been dirtied yet.
*/
WALInsertLockAcquireExclusive();
- if (XLogCtl->Insert.lastBackupStart < state->startpoint)
+ if (XLogCtl->Insert.lastBackupStart < backup_state->startpoint)
{
- XLogCtl->Insert.lastBackupStart = state->startpoint;
+ XLogCtl->Insert.lastBackupStart = backup_state->startpoint;
gotUniqueStartpoint = true;
}
WALInsertLockRelease();
@@ -8529,11 +8529,11 @@ do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
}
FreeDir(tblspcdir);
- state->starttime = (pg_time_t) time(NULL);
+ backup_state->starttime = (pg_time_t) time(NULL);
}
PG_END_ENSURE_ERROR_CLEANUP(pg_backup_start_callback, (Datum) 0);
- state->started_in_recovery = backup_started_in_recovery;
+ backup_state->started_in_recovery = backup_started_in_recovery;
/*
* Mark that the start phase has correctly finished for the backup.
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index dce265098e..9e01146c6c 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -279,7 +279,7 @@ typedef enum SessionBackupState
} SessionBackupState;
extern void do_pg_backup_start(const char *backupidstr, bool fast,
- List **tablespaces, BackupState *state,
+ List **tablespaces, BackupState *backup_state,
StringInfo tblspcmapfile);
extern void do_pg_backup_stop(BackupState *state, bool waitforarchive);
extern void do_pg_abort_backup(int code, Datum arg);
On Tue, Sep 27, 2022 at 1:54 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
This commit introduced BackupState struct. The comment of
do_pg_backup_start says that:* It fills in backup_state with the information required for the backup,
And the parameters are:
do_pg_backup_start(const char *backupidstr, bool fast, List **tablespaces,
BackupState *state, StringInfo tblspcmapfile)So backup_state is different from both the type BackupState and the
parameter state. I find it annoying. Don't we either rename the
parameter or fix the comment?The parameter "state" sounds a bit too generic. So I prefer to rename
the parameter to backup_state, as the attached.What do you think about this?
-1 from me. We have the function context and the structure name there
to represent that the parameter name 'state' is actually 'backup
state'. I don't think we gain anything here. Whenever the BackupState
is used away from any function, the variable name backup_state is
used, static variable in xlogfuncs.c
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
At Tue, 27 Sep 2022 14:03:24 +0530, Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> wrote in
On Tue, Sep 27, 2022 at 1:54 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:What do you think about this?
-1 from me. We have the function context and the structure name there
to represent that the parameter name 'state' is actually 'backup
state'. I don't think we gain anything here. Whenever the BackupState
is used away from any function, the variable name backup_state is
used, static variable in xlogfuncs.c
There's no shadowing caused by the change. If we mind the same
variable names between files, we could rename backup_state in
xlogfuncs.c to process_backup_state or session_backup_state.
If this is still unacceptable, I propose to change the comment. (I
found that the previous patch forgets about do_pg_backup_stop())
- * It fills in backup_state with the information required for the backup,
+ * It fills in the parameter "state" with the information required for the backup,
(This is following the notation just above)
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
On Tue, Sep 27, 2022 at 2:20 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:
-1 from me. We have the function context and the structure name there
to represent that the parameter name 'state' is actually 'backup
state'. I don't think we gain anything here. Whenever the BackupState
is used away from any function, the variable name backup_state is
used, static variable in xlogfuncs.cThere's no shadowing caused by the change. If we mind the same
variable names between files, we could rename backup_state in
xlogfuncs.c to process_backup_state or session_backup_state.
-1.
If this is still unacceptable, I propose to change the comment. (I
found that the previous patch forgets about do_pg_backup_stop())- * It fills in backup_state with the information required for the backup, + * It fills in the parameter "state" with the information required for the backup,
+1. There's another place that uses backup_state in the comments. I
modified that as well. Please see the attached patch.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
v1-0001-Adjust-BackupState-comments-to-not-use-backup_sta.patchapplication/x-patch; name=v1-0001-Adjust-BackupState-comments-to-not-use-backup_sta.patchDownload
From b66c67712f3499259427c3ea32d102ac802941ac Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Date: Tue, 27 Sep 2022 09:36:32 +0000
Subject: [PATCH v1] Adjust BackupState comments to not use backup_state
---
src/backend/access/transam/xlog.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1dd6df0fe1..675f851daa 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -8256,9 +8256,10 @@ issue_xlog_fsync(int fd, XLogSegNo segno, TimeLineID tli)
* symlinks while extracting files from tar. However for consistency and
* platform-independence, we do it the same way everywhere.
*
- * It fills in backup_state with the information required for the backup,
- * such as the minimum WAL location that must be present to restore from
- * this backup (starttli) and the corresponding timeline ID (starttli).
+ * It fills in the parameter "state" with the information required for the
+ * backup, such as the minimum WAL location that must be present to restore
+ * from this backup (starttli) and the corresponding timeline ID (starttli),
+ * etc.
*
* Every successfully started backup must be stopped by calling
* do_pg_backup_stop() or do_pg_abort_backup(). There can be many
@@ -8574,8 +8575,9 @@ get_backup_status(void)
* file (if required), resets sessionBackupState and so on. It can optionally
* wait for WAL segments to be archived.
*
- * backup_state is filled with the information necessary to restore from this
- * backup with its stop LSN (stoppoint), its timeline ID (stoptli), etc.
+ * It fills in the parameter "state" with the information necessary to restore
+ * from this backup with its stop LSN (stoppoint), its timeline ID (stoptli),
+ * etc.
*
* It is the responsibility of the caller of this function to verify the
* permissions of the calling user!
--
2.34.1
On Tue, Sep 27, 2022 at 03:11:54PM +0530, Bharath Rupireddy wrote:
On Tue, Sep 27, 2022 at 2:20 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:If this is still unacceptable, I propose to change the comment. (I
found that the previous patch forgets about do_pg_backup_stop())- * It fills in backup_state with the information required for the backup, + * It fills in the parameter "state" with the information required for the backup,+1. There's another place that uses backup_state in the comments. I
modified that as well. Please see the attached patch.
Thanks, fixed the comments. I have let the variable names as they are
now in the code, as both are backup-related code paths so it is IMO
clear that the state is linked to a backup.
--
Michael
At Wed, 28 Sep 2022 10:09:39 +0900, Michael Paquier <michael@paquier.xyz> wrote in
On Tue, Sep 27, 2022 at 03:11:54PM +0530, Bharath Rupireddy wrote:
On Tue, Sep 27, 2022 at 2:20 PM Kyotaro Horiguchi
<horikyota.ntt@gmail.com> wrote:If this is still unacceptable, I propose to change the comment. (I
found that the previous patch forgets about do_pg_backup_stop())- * It fills in backup_state with the information required for the backup, + * It fills in the parameter "state" with the information required for the backup,+1. There's another place that uses backup_state in the comments. I
modified that as well. Please see the attached patch.Thanks, fixed the comments. I have let the variable names as they are
now in the code, as both are backup-related code paths so it is IMO
clear that the state is linked to a backup.
Thanks! I'm fine with that.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center