Minor changes to Recovery related code
I'd like to make the following changes to recovery related code over the
next few days/weeks. If anybody insists I do this by freeze or not at
all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
rather take a bit more time and do this in one drop and there are some
code dependencies between these changes and other patches from
Koichi-san and myself.
1. Current xlog should be archived at shutdown for smart shutdown
- check that archiver is active prior to shutdown checkpoint request
- if (shutdown checkpoint && XLogArchivingActive()) RequestXLogSwitch()
- for smart shutdown, have archiver complete its work before exiting
2. pg_stop_backup() should wait until all archive files are safely
archived before returning
3. Need a %r parameter for restore_command, to allow the restore command
be passed the name of the file containing the last restartpoint. This
will allow the restore command to clean down old archive files more
safely/cleanly in Warm Standby operation.
- change to pg_standby to accept the parameter and use %r rather than -k
parameter
4. Add an option to pg_standby to have it complete all outstanding
archive files after it has been triggered, minimising data loss at the
slight expense of cut-over time
5. log_restartpoint option in recovery.conf
LOG each restorepoint, so can understand whether restartable recovery
will be effective or not
6. refactor recovery.conf so that it uses a GUC-like parser
7. refactor all xlog _desc routines into one module, so these can be
more easily used by xlogviewer utility
8. get xlogviewer utility a freshen-up so it can be part of main
release, possibly including further refactoring of xlog.c
9. Another round of doc updates to highlight the use of pg_standby and
Koichi-san's work. I think Doug Knight might have some additional code
examples to include as well, from previous discussions.
10. Changes to ensure WAL-avoiding operations and hot backups cannot be
executed simultaneously. One of these two options, ISTM:
a) Following a change to archive_command while server is running. Log
the xid of the WAL-avoiding operations when they start and have
pg_start_backup() wait for those xids to complete before continuing.
b) Introduce a new parameter, archive_mode = on | off that can only be
set at server start. If archive_mode = on then XLogArchivingActive();
archiving only takes place when archive_command is not ''. This allows
archive_command to be changed while server running, yet without any
danger from WAL-avoiding operations.
[7 & 8 would be complete by about 5-6 weeks from now. Others much
earlier]
Comments?
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
"Simon Riggs" <simon@2ndquadrant.com> writes:
I'd like to make the following changes to recovery related code over the
next few days/weeks. If anybody insists I do this by freeze or not at
all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
rather take a bit more time and do this in one drop and there are some
code dependencies between these changes and other patches from
Koichi-san and myself.
Well, I've got a proposal for a pg_proc change that I haven't even started
coding yet, so personally I won't hold you to having the patch submitted
as long as the design is agreed to before feature freeze. However:
2. pg_stop_backup() should wait until all archive files are safely
archived before returning
Not sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.
6. refactor recovery.conf so that it uses a GUC-like parser
I would suggest that after feature freeze is not the time for code
beautification efforts like this. (I rather doubt that it's worth doing
at all actually, but definitely not now.)
7. refactor all xlog _desc routines into one module, so these can be
more easily used by xlogviewer utility
Even more so.
8. get xlogviewer utility a freshen-up so it can be part of main
release, possibly including further refactoring of xlog.c
This is not happening for 8.3, either.
10. Changes to ensure WAL-avoiding operations and hot backups cannot be
executed simultaneously. One of these two options, ISTM:
b) Introduce a new parameter, archive_mode = on | off that can only be
set at server start. If archive_mode = on then XLogArchivingActive();
archiving only takes place when archive_command is not ''. This allows
archive_command to be changed while server running, yet without any
danger from WAL-avoiding operations.
I think I'd go with (b) since a lot of people felt it should've been
like that from the beginning, and found the magic "empty string"
behavior confusing.
[7 & 8 would be complete by about 5-6 weeks from now. Others much
earlier]
We are hoping to go beta in less time than that. While I'm willing to
cut a little slack, anything you can't submit before about mid-April
is not going to make it into 8.3.
regards, tom lane
On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
"Simon Riggs" <simon@2ndquadrant.com> writes:
I'd like to make the following changes to recovery related code over the
next few days/weeks. If anybody insists I do this by freeze or not at
all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
rather take a bit more time and do this in one drop and there are some
code dependencies between these changes and other patches from
Koichi-san and myself.Well, I've got a proposal for a pg_proc change that I haven't even started
coding yet, so personally I won't hold you to having the patch submitted
as long as the design is agreed to before feature freeze. However:
Cool
2. pg_stop_backup() should wait until all archive files are safely
archived before returningNot sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.
There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.
We need a way to plug that small hole.
I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.
6. refactor recovery.conf so that it uses a GUC-like parser
I would suggest that after feature freeze is not the time for code
beautification efforts like this. (I rather doubt that it's worth doing
at all actually, but definitely not now.)
OK
7. refactor all xlog _desc routines into one module, so these can be
more easily used by xlogviewer utilityEven more so.
OK
8. get xlogviewer utility a freshen-up so it can be part of main
release, possibly including further refactoring of xlog.cThis is not happening for 8.3, either.
OK
10. Changes to ensure WAL-avoiding operations and hot backups cannot be
executed simultaneously. One of these two options, ISTM:
b) Introduce a new parameter, archive_mode = on | off that can only be
set at server start. If archive_mode = on then XLogArchivingActive();
archiving only takes place when archive_command is not ''. This allows
archive_command to be changed while server running, yet without any
danger from WAL-avoiding operations.I think I'd go with (b) since a lot of people felt it should've been
like that from the beginning, and found the magic "empty string"
behavior confusing.
OK
We are hoping to go beta in less time than that. While I'm willing to
cut a little slack, anything you can't submit before about mid-April
is not going to make it into 8.3.
No probs, you struck a line through the more time consuming items. :-)
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
Simon Riggs wrote:
On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
"Simon Riggs" <simon@2ndquadrant.com> writes:
2. pg_stop_backup() should wait until all archive files are safely
archived before returningNot sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.We need a way to plug that small hole.
I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.
I fear that ending sucessfully despite having not archived all wals
will make this feature less worthwile. If a dba knows what he is
doing, he can code a perfectly safe backup script using 8.2 too.
He'll just have to check the current wal position after pg_stop_backup(),
(There is a function for that, right?), and wait until the corresponding
wal was archived.
In realitly, however, I feare that most people will just create a script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
the return value of pgsql. I believe that those are the people who would
really benefit from a pg_stop_backup() that waits for archiving to complete.
But they probably won't check for WARNINGs.
Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.
Anyway, just my 0.02 eurocents, maybe I'm totally mistaken about the
postgresql dba's out there...
greetings, Florian Pflug
On Mar 30, 2007, at 5:51 PM, Florian G. Pflug wrote:
In realitly, however, I feare that most people will just create a
script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and
check
the return value of pgsql. I believe that those are the people who
would
really benefit from a pg_stop_backup() that waits for archiving to
complete.
But they probably won't check for WARNINGs.Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.
I agree; people are far more likely to take the lazy way out, so it'd
be good to protect against that. Esp. since statement_timeout
provides a means to get the other behavior.
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
Simon Riggs wrote:
On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
"Simon Riggs" <simon@2ndquadrant.com> writes:
2. pg_stop_backup() should wait until all archive files are safely
archived before returningNot sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.We need a way to plug that small hole.
I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.I fear that ending sucessfully despite having not archived all wals
will make this feature less worthwile. If a dba knows what he is
doing, he can code a perfectly safe backup script using 8.2 too.
He'll just have to check the current wal position after pg_stop_backup(),
(There is a function for that, right?), and wait until the corresponding
wal was archived.In realitly, however, I feare that most people will just create a script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
the return value of pgsql. I believe that those are the people who would
really benefit from a pg_stop_backup() that waits for archiving to complete.
But they probably won't check for WARNINGs.Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.
I've just come up against this problem again, so I think it is a must
fix for this release. Other problems exist also, mentioned on separate
threads.
We have a number of problems surrounding pg_stop_backup/shutdown:
1. pg_stop_backup() currently returns before the WAL file containing the
last change is correctly archived. That is a small hole, but one that is
exposed when people write test scripts that immediately shutdown the
database after issuing pg_stop_backup(). It doesn't make much sense to
shutdown immediately after a hot backup, but it should still work
sensibly.
2. We've also had problems caused by making the archiver wait until all
WAL files are archived. If there is a backlog for some reason and the
DBA issues a restart (i.e. stop and immediate restart) then making the
archiver loop while it tries (and possibly fails) to archive all files
would cause an outage. Avoiding this is why we do the current
get-out-fast approach.
There are some sub scenarios:
a) there is a backlog of WAL files, but no error has occurred on the
*last* file (we might have just fixed a problem).
b) there is a backlog of WAL files, but an error is causing a retry of
the last file.
My proposal is for us to record somewhere other than the logs that a
failure to archive has occurred and is being retried. Failure to archive
will be recorded in the archive_status directory as an additional file
called archive_error, which will be deleted in the case of archive
success and created in the case of archive error. This maintains
archiver's lack of attachment to shared memory and general simplicity of
design.
- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.
- new function: pg_stop_backup_nowait() return immediately without
waiting for archive, the same as the current pg_stop_backup()
- new function: pg_stop_backup_wait(int seconds) wait until either an
archival fails or the ending WAL file is archived, with a max wait as
specified. wait=0 means wait until archive errors are resolved.
Alternatives?
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
We have a number of problems surrounding pg_stop_backup/shutdown:
I'll start coding up my proposals, in the absence of alternate views.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
This has been saved for the 8.4 release:
http://momjian.postgresql.org/cgi-bin/pgpatches_hold
---------------------------------------------------------------------------
Simon Riggs wrote:
On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
Simon Riggs wrote:
On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
"Simon Riggs" <simon@2ndquadrant.com> writes:
2. pg_stop_backup() should wait until all archive files are safely
archived before returningNot sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.We need a way to plug that small hole.
I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.I fear that ending sucessfully despite having not archived all wals
will make this feature less worthwile. If a dba knows what he is
doing, he can code a perfectly safe backup script using 8.2 too.
He'll just have to check the current wal position after pg_stop_backup(),
(There is a function for that, right?), and wait until the corresponding
wal was archived.In realitly, however, I feare that most people will just create a script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
the return value of pgsql. I believe that those are the people who would
really benefit from a pg_stop_backup() that waits for archiving to complete.
But they probably won't check for WARNINGs.Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.I've just come up against this problem again, so I think it is a must
fix for this release. Other problems exist also, mentioned on separate
threads.We have a number of problems surrounding pg_stop_backup/shutdown:
1. pg_stop_backup() currently returns before the WAL file containing the
last change is correctly archived. That is a small hole, but one that is
exposed when people write test scripts that immediately shutdown the
database after issuing pg_stop_backup(). It doesn't make much sense to
shutdown immediately after a hot backup, but it should still work
sensibly.2. We've also had problems caused by making the archiver wait until all
WAL files are archived. If there is a backlog for some reason and the
DBA issues a restart (i.e. stop and immediate restart) then making the
archiver loop while it tries (and possibly fails) to archive all files
would cause an outage. Avoiding this is why we do the current
get-out-fast approach.
There are some sub scenarios:
a) there is a backlog of WAL files, but no error has occurred on the
*last* file (we might have just fixed a problem).
b) there is a backlog of WAL files, but an error is causing a retry of
the last file.My proposal is for us to record somewhere other than the logs that a
failure to archive has occurred and is being retried. Failure to archive
will be recorded in the archive_status directory as an additional file
called archive_error, which will be deleted in the case of archive
success and created in the case of archive error. This maintains
archiver's lack of attachment to shared memory and general simplicity of
design.- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.- new function: pg_stop_backup_nowait() return immediately without
waiting for archive, the same as the current pg_stop_backup()- new function: pg_stop_backup_wait(int seconds) wait until either an
archival fails or the ending WAL file is archived, with a max wait as
specified. wait=0 means wait until archive errors are resolved.Alternatives?
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Follow-up during March 2008 CommitFest
On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
Simon Riggs wrote:
On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
"Simon Riggs" <simon@2ndquadrant.com> writes:
2. pg_stop_backup() should wait until all archive files are safely
archived before returningNot sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.We need a way to plug that small hole.
I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.I fear that ending sucessfully despite having not archived all wals
will make this feature less worthwile. If a dba knows what he is
doing, he can code a perfectly safe backup script using 8.2 too.
He'll just have to check the current wal position after pg_stop_backup(),
(There is a function for that, right?), and wait until the corresponding
wal was archived.In realitly, however, I feare that most people will just create a script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
the return value of pgsql. I believe that those are the people who would
really benefit from a pg_stop_backup() that waits for archiving to complete.
But they probably won't check for WARNINGs.Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.I've just come up against this problem again, so I think it is a must
fix for this release. Other problems exist also, mentioned on separate
threads.We have a number of problems surrounding pg_stop_backup/shutdown:
1. pg_stop_backup() currently returns before the WAL file containing the
last change is correctly archived. That is a small hole, but one that is
exposed when people write test scripts that immediately shutdown the
database after issuing pg_stop_backup(). It doesn't make much sense to
shutdown immediately after a hot backup, but it should still work
sensibly.2. We've also had problems caused by making the archiver wait until all
WAL files are archived. If there is a backlog for some reason and the
DBA issues a restart (i.e. stop and immediate restart) then making the
archiver loop while it tries (and possibly fails) to archive all files
would cause an outage. Avoiding this is why we do the current
get-out-fast approach.
There are some sub scenarios:
a) there is a backlog of WAL files, but no error has occurred on the
*last* file (we might have just fixed a problem).
b) there is a backlog of WAL files, but an error is causing a retry of
the last file.My proposal is for us to record somewhere other than the logs that a
failure to archive has occurred and is being retried. Failure to archive
will be recorded in the archive_status directory as an additional file
called archive_error, which will be deleted in the case of archive
success and created in the case of archive error. This maintains
archiver's lack of attachment to shared memory and general simplicity of
design.- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.
The most straightforward thing is to make pg_stop_backup() wait as
described above.
If people want it to timeout, they can use a statement_timeout as
suggested by Florian.
This can be implemented by having the function poll in an infinite loop
for archive_status/LOG.done. Also, as Florian suggests, we should have
it output a WARNING message every 60 seconds.
I'll write a patch now.
- new function: pg_stop_backup_nowait() return immediately without
waiting for archive, the same as the current pg_stop_backup()- new function: pg_stop_backup_wait(int seconds) wait until either an
archival fails or the ending WAL file is archived, with a max wait as
specified. wait=0 means wait until archive errors are resolved.
So I won't do these.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
On Thu, 2008-03-27 at 17:34 +0000, Simon Riggs wrote:
Follow-up during March 2008 CommitFest
On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.The most straightforward thing is to make pg_stop_backup() wait as
described above.If people want it to timeout, they can use a statement_timeout as
suggested by Florian.This can be implemented by having the function poll in an infinite loop
for archive_status/LOG.done. Also, as Florian suggests, we should have
it output a WARNING message every 60 seconds.I'll write a patch now.
Patch to implement waiting pg_stop_backup().
Tested and ready to apply, includes docs.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
Attachments:
stop_backup_wait.v1.patchtext/x-patch; charset=UTF-8; name=stop_backup_wait.v1.patchDownload
Index: doc/src/sgml/backup.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/backup.sgml,v
retrieving revision 2.116
diff -c -r2.116 backup.sgml
*** doc/src/sgml/backup.sgml 28 Mar 2008 15:00:28 -0000 2.116
--- doc/src/sgml/backup.sgml 31 Mar 2008 14:11:59 -0000
***************
*** 761,772 ****
<para>
Once the WAL segment files used during the backup are archived, you are
done. The file identified by <function>pg_stop_backup</>'s result is
! the last segment that needs to be archived to complete the backup.
Archival of these files will happen automatically, since you have
already configured <varname>archive_command</>. In many cases, this
happens fairly quickly, but you are advised to monitor your archival
! system to ensure this has taken place so that you can be certain you
! have a complete backup.
</para>
</listitem>
</orderedlist>
--- 761,779 ----
<para>
Once the WAL segment files used during the backup are archived, you are
done. The file identified by <function>pg_stop_backup</>'s result is
! the last segment that is required to form a complete set of backup files.
! <function>pg_stop_backup</> does not return until the last segment has
! been archived.
Archival of these files will happen automatically, since you have
already configured <varname>archive_command</>. In many cases, this
happens fairly quickly, but you are advised to monitor your archival
! system to ensure there is no delay.
! If the archival process has fallen significantly behind
! because of failures of the archive command then we will keep waiting
! and retrying until the archive succeeds and the backup is complete.
! If you wish to place a time limit on the execution of
! <function>pg_stop_backup</> then you should set an appropriate
! <varname>statement_timeout</varname>.
</para>
</listitem>
</orderedlist>
***************
*** 1044,1050 ****
<note>
<para>
The stop point must be after the ending time of the base backup, i.e.,
! the time of <function>pg_stop_backup</>. You cannot use a base backup
to recover to a time when that backup was still going on. (To
recover to such a time, you must go back to your previous base backup
and roll forward from there.)
--- 1051,1057 ----
<note>
<para>
The stop point must be after the ending time of the base backup, i.e.,
! the end time of <function>pg_stop_backup</>. You cannot use a base backup
to recover to a time when that backup was still going on. (To
recover to such a time, you must go back to your previous base backup
and roll forward from there.)
***************
*** 1322,1327 ****
--- 1329,1335 ----
After the backup the switch file is removed. Archived WAL files are
then added to the backup so that both base backup and all required
WAL files are part of the same <application>tar</> file.
+ Please remember to add error handling to your backup scripts.
</para>
</sect3>
Index: src/backend/access/transam/xlog.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/backend/access/transam/xlog.c,v
retrieving revision 1.295
diff -c -r1.295 xlog.c
*** src/backend/access/transam/xlog.c 25 Mar 2008 22:42:42 -0000 1.295
--- src/backend/access/transam/xlog.c 1 Apr 2008 16:08:07 -0000
***************
*** 382,388 ****
static void XLogArchiveNotify(const char *xlog);
static void XLogArchiveNotifySeg(uint32 log, uint32 seg);
! static bool XLogArchiveCheckDone(const char *xlog);
static void XLogArchiveCleanup(const char *xlog);
static void readRecoveryCommandFile(void);
static void exitArchiveRecovery(TimeLineID endTLI,
--- 382,388 ----
static void XLogArchiveNotify(const char *xlog);
static void XLogArchiveNotifySeg(uint32 log, uint32 seg);
! static bool XLogArchiveCheckDone(const char *xlog, bool create_if_missing);
static void XLogArchiveCleanup(const char *xlog);
static void readRecoveryCommandFile(void);
static void exitArchiveRecovery(TimeLineID endTLI,
***************
*** 1128,1134 ****
* create <XLOG>.ready fails, we'll retry during subsequent checkpoints.
*/
static bool
! XLogArchiveCheckDone(const char *xlog)
{
char archiveStatusPath[MAXPGPATH];
struct stat stat_buf;
--- 1128,1134 ----
* create <XLOG>.ready fails, we'll retry during subsequent checkpoints.
*/
static bool
! XLogArchiveCheckDone(const char *xlog, bool create_if_missing)
{
char archiveStatusPath[MAXPGPATH];
struct stat stat_buf;
***************
*** 1153,1159 ****
return true;
/* Retry creation of the .ready file */
! XLogArchiveNotify(xlog);
return false;
}
--- 1153,1161 ----
return true;
/* Retry creation of the .ready file */
! if (create_if_missing)
! XLogArchiveNotify(xlog);
!
return false;
}
***************
*** 2704,2710 ****
strspn(xlde->d_name, "0123456789ABCDEF") == 24 &&
strcmp(xlde->d_name + 8, lastoff + 8) <= 0)
{
! if (XLogArchiveCheckDone(xlde->d_name))
{
snprintf(path, MAXPGPATH, XLOGDIR "/%s", xlde->d_name);
--- 2706,2712 ----
strspn(xlde->d_name, "0123456789ABCDEF") == 24 &&
strcmp(xlde->d_name + 8, lastoff + 8) <= 0)
{
! if (XLogArchiveCheckDone(xlde->d_name, true))
{
snprintf(path, MAXPGPATH, XLOGDIR "/%s", xlde->d_name);
***************
*** 2771,2777 ****
strcmp(xlde->d_name + strlen(xlde->d_name) - strlen(".backup"),
".backup") == 0)
{
! if (XLogArchiveCheckDone(xlde->d_name))
{
ereport(DEBUG2,
(errmsg("removing transaction log backup history file \"%s\"",
--- 2773,2779 ----
strcmp(xlde->d_name + strlen(xlde->d_name) - strlen(".backup"),
".backup") == 0)
{
! if (XLogArchiveCheckDone(xlde->d_name, true))
{
ereport(DEBUG2,
(errmsg("removing transaction log backup history file \"%s\"",
***************
*** 6556,6561 ****
--- 6558,6565 ----
FILE *fp;
char ch;
int ich;
+ int seconds_before_warning;
+ int waits = 0;
if (!superuser())
ereport(ERROR,
***************
*** 6660,6665 ****
--- 6664,6702 ----
CleanupBackupHistory();
/*
+ * Wait until the history file has been archived. We assume that the
+ * alphabetic sorting property of the WAL files ensures the last WAL
+ * file is guaranteed archived by the time the history file is archived.
+ *
+ * We wait for ever, since archive_command is supposed to work and
+ * we assume the admin wanted his backup to work completely. If you
+ * don't wish to wait, you can SET statement_timeout = xx;
+ *
+ * If the status file is missing, we assume that is because it was
+ * set to .ready before we slept, then while asleep it has been set
+ * to .done and then removed by a concurrent checkpoint.
+ */
+ BackupHistoryFileName(histfilepath, ThisTimeLineID, _logId, _logSeg,
+ startpoint.xrecoff % XLogSegSize);
+
+ seconds_before_warning = 60;
+ waits = 0;
+
+ while (!XLogArchiveCheckDone(histfilepath, false))
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ pg_usleep(1000000L);
+
+ if (++waits >= seconds_before_warning)
+ {
+ seconds_before_warning *= 2; /* This wraps in >10 years... */
+ elog(WARNING, "pg_stop_backup() waiting for archive to complete "
+ "(%d seconds delay)", waits);
+ }
+ }
+
+ /*
* We're done. As a convenience, return the ending WAL location.
*/
snprintf(stopxlogfilename, sizeof(stopxlogfilename), "%X/%X",
Nice, applied. I only modified some of the documentation wording.
I was a little worried that statement_timeout might cancel
pg_stop_backup() _before_ it had gotten to waiting for the archive logs
to be saved, but based on what little code there is before that block, I
think we are OK.
---------------------------------------------------------------------------
Simon Riggs wrote:
On Thu, 2008-03-27 at 17:34 +0000, Simon Riggs wrote:
Follow-up during March 2008 CommitFest
On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.The most straightforward thing is to make pg_stop_backup() wait as
described above.If people want it to timeout, they can use a statement_timeout as
suggested by Florian.This can be implemented by having the function poll in an infinite loop
for archive_status/LOG.done. Also, as Florian suggests, we should have
it output a WARNING message every 60 seconds.I'll write a patch now.
Patch to implement waiting pg_stop_backup().
Tested and ready to apply, includes docs.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.comPostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
[ Attachment, skipping... ]
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Attachments:
/rtmp/difftext/x-diffDownload
Index: doc/src/sgml/backup.sgml
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/backup.sgml,v
retrieving revision 2.116
diff -c -c -r2.116 backup.sgml
*** doc/src/sgml/backup.sgml 28 Mar 2008 15:00:28 -0000 2.116
--- doc/src/sgml/backup.sgml 5 Apr 2008 01:28:09 -0000
***************
*** 761,772 ****
<para>
Once the WAL segment files used during the backup are archived, you are
done. The file identified by <function>pg_stop_backup</>'s result is
! the last segment that needs to be archived to complete the backup.
! Archival of these files will happen automatically, since you have
! already configured <varname>archive_command</>. In many cases, this
! happens fairly quickly, but you are advised to monitor your archival
! system to ensure this has taken place so that you can be certain you
! have a complete backup.
</para>
</listitem>
</orderedlist>
--- 761,779 ----
<para>
Once the WAL segment files used during the backup are archived, you are
done. The file identified by <function>pg_stop_backup</>'s result is
! the last segment that is required to form a complete set of backup files.
! <function>pg_stop_backup</> does not return until the last segment has
! been archived.
! Archiving of these files happens automatically since you have
! already configured <varname>archive_command</>. In most cases this
! happens quickly, but you are advised to monitor your archive
! system to ensure there are no delays.
! If the archive process has fallen behind
! because of failures of the archive command, it will keep retrying
! until the archive succeeds and the backup is complete.
! If you wish to place a time limit on the execution of
! <function>pg_stop_backup</>, set an appropriate
! <varname>statement_timeout</varname> value.
</para>
</listitem>
</orderedlist>
***************
*** 1044,1050 ****
<note>
<para>
The stop point must be after the ending time of the base backup, i.e.,
! the time of <function>pg_stop_backup</>. You cannot use a base backup
to recover to a time when that backup was still going on. (To
recover to such a time, you must go back to your previous base backup
and roll forward from there.)
--- 1051,1057 ----
<note>
<para>
The stop point must be after the ending time of the base backup, i.e.,
! the end time of <function>pg_stop_backup</>. You cannot use a base backup
to recover to a time when that backup was still going on. (To
recover to such a time, you must go back to your previous base backup
and roll forward from there.)
***************
*** 1322,1327 ****
--- 1329,1335 ----
After the backup the switch file is removed. Archived WAL files are
then added to the backup so that both base backup and all required
WAL files are part of the same <application>tar</> file.
+ Please remember to add error handling to your backup scripts.
</para>
</sect3>
Index: src/backend/access/transam/xlog.c
===================================================================
RCS file: /cvsroot/pgsql/src/backend/access/transam/xlog.c,v
retrieving revision 1.295
diff -c -c -r1.295 xlog.c
*** src/backend/access/transam/xlog.c 25 Mar 2008 22:42:42 -0000 1.295
--- src/backend/access/transam/xlog.c 5 Apr 2008 01:28:09 -0000
***************
*** 382,388 ****
static void XLogArchiveNotify(const char *xlog);
static void XLogArchiveNotifySeg(uint32 log, uint32 seg);
! static bool XLogArchiveCheckDone(const char *xlog);
static void XLogArchiveCleanup(const char *xlog);
static void readRecoveryCommandFile(void);
static void exitArchiveRecovery(TimeLineID endTLI,
--- 382,388 ----
static void XLogArchiveNotify(const char *xlog);
static void XLogArchiveNotifySeg(uint32 log, uint32 seg);
! static bool XLogArchiveCheckDone(const char *xlog, bool create_if_missing);
static void XLogArchiveCleanup(const char *xlog);
static void readRecoveryCommandFile(void);
static void exitArchiveRecovery(TimeLineID endTLI,
***************
*** 1128,1134 ****
* create <XLOG>.ready fails, we'll retry during subsequent checkpoints.
*/
static bool
! XLogArchiveCheckDone(const char *xlog)
{
char archiveStatusPath[MAXPGPATH];
struct stat stat_buf;
--- 1128,1134 ----
* create <XLOG>.ready fails, we'll retry during subsequent checkpoints.
*/
static bool
! XLogArchiveCheckDone(const char *xlog, bool create_if_missing)
{
char archiveStatusPath[MAXPGPATH];
struct stat stat_buf;
***************
*** 1153,1159 ****
return true;
/* Retry creation of the .ready file */
! XLogArchiveNotify(xlog);
return false;
}
--- 1153,1161 ----
return true;
/* Retry creation of the .ready file */
! if (create_if_missing)
! XLogArchiveNotify(xlog);
!
return false;
}
***************
*** 2704,2710 ****
strspn(xlde->d_name, "0123456789ABCDEF") == 24 &&
strcmp(xlde->d_name + 8, lastoff + 8) <= 0)
{
! if (XLogArchiveCheckDone(xlde->d_name))
{
snprintf(path, MAXPGPATH, XLOGDIR "/%s", xlde->d_name);
--- 2706,2712 ----
strspn(xlde->d_name, "0123456789ABCDEF") == 24 &&
strcmp(xlde->d_name + 8, lastoff + 8) <= 0)
{
! if (XLogArchiveCheckDone(xlde->d_name, true))
{
snprintf(path, MAXPGPATH, XLOGDIR "/%s", xlde->d_name);
***************
*** 2771,2777 ****
strcmp(xlde->d_name + strlen(xlde->d_name) - strlen(".backup"),
".backup") == 0)
{
! if (XLogArchiveCheckDone(xlde->d_name))
{
ereport(DEBUG2,
(errmsg("removing transaction log backup history file \"%s\"",
--- 2773,2779 ----
strcmp(xlde->d_name + strlen(xlde->d_name) - strlen(".backup"),
".backup") == 0)
{
! if (XLogArchiveCheckDone(xlde->d_name, true))
{
ereport(DEBUG2,
(errmsg("removing transaction log backup history file \"%s\"",
***************
*** 6556,6561 ****
--- 6558,6565 ----
FILE *fp;
char ch;
int ich;
+ int seconds_before_warning;
+ int waits = 0;
if (!superuser())
ereport(ERROR,
***************
*** 6660,6665 ****
--- 6664,6702 ----
CleanupBackupHistory();
/*
+ * Wait until the history file has been archived. We assume that the
+ * alphabetic sorting property of the WAL files ensures the last WAL
+ * file is guaranteed archived by the time the history file is archived.
+ *
+ * We wait forever, since archive_command is supposed to work and
+ * we assume the admin wanted his backup to work completely. If you
+ * don't wish to wait, you can SET statement_timeout = xx;
+ *
+ * If the status file is missing, we assume that is because it was
+ * set to .ready before we slept, then while asleep it has been set
+ * to .done and then removed by a concurrent checkpoint.
+ */
+ BackupHistoryFileName(histfilepath, ThisTimeLineID, _logId, _logSeg,
+ startpoint.xrecoff % XLogSegSize);
+
+ seconds_before_warning = 60;
+ waits = 0;
+
+ while (!XLogArchiveCheckDone(histfilepath, false))
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ pg_usleep(1000000L);
+
+ if (++waits >= seconds_before_warning)
+ {
+ seconds_before_warning *= 2; /* This wraps in >10 years... */
+ elog(WARNING, "pg_stop_backup() waiting for archive to complete "
+ "(%d seconds delay)", waits);
+ }
+ }
+
+ /*
* We're done. As a convenience, return the ending WAL location.
*/
snprintf(stopxlogfilename, sizeof(stopxlogfilename), "%X/%X",