Online base backup from the hot-standby

On 14.06.2011 09:03, Jun Ishiduka wrote:

( Quotation from
http://archives.postgresql.org/pgsql-hackers/2011-05/msg01396.php )

STEP1: Make startup process to acquire backup-end-position from
not only backup-end record but also backup-history-file .
* startup process allows to acquire backup-end-position
from backup-history-file .

I have created a patch to the above-mentioned content.

I still think that's headed in the wrong direction.
(http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php)

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

I still think that's headed in the wrong direction.
(http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php)

Please check these mails, and teach the reason for content of the wrong
direction.
(http://archives.postgresql.org/pgsql-hackers/2011-06/msg00209.php)
(http://archives.postgresql.org/pgsql-hackers/2011-05/msg01566.php)

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

On 11-06-14 02:52 AM, Jun Ishiduka wrote:

I still think that's headed in the wrong direction.
(http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php)

Please check these mails, and teach the reason for content of the wrong
direction.
(http://archives.postgresql.org/pgsql-hackers/2011-06/msg00209.php)
(http://archives.postgresql.org/pgsql-hackers/2011-05/msg01566.php)

Jun, I've been reviewing these threads as a start to reviewing your
patch (I haven't yet looked at the patch).

I *think* the concern is that

1) Today you can do a backup by just calling pg_start_backup('x'); copy
the data directory and
pg_stop_backup(); You do not need to use pg_basebackup to create a
backup. The solution you are proposing would require pg_basebackup to be
used to build backups from standby servers.

2) If I run pg_basebackup but do not specify '-x' then no pg_xlog
segments are included in the output. The relevant pg_xlog segments are
in the archive from the master. I can see situations where you are
already copying the archive to the remote site that the new standby will
be created in so you don't want to have to copy the pg_xlog segments
twice over your network.

What Heikki is proposing will work both when you aren't using
pg_basebackup (as long the output of pg_stop_backup() is somehow
captured in a way that it can be read) and will also work with
pg_basebackup when '-x' isn't specified.

Steve

1) Today you can do a backup by just calling pg_start_backup('x'); copy
the data directory and
pg_stop_backup(); You do not need to use pg_basebackup to create a
backup. The solution you are proposing would require pg_basebackup to be
used to build backups from standby servers.

YES.

2) If I run pg_basebackup but do not specify '-x' then no pg_xlog
segments are included in the output. The relevant pg_xlog segments are
in the archive from the master. I can see situations where you are
already copying the archive to the remote site that the new standby will
be created in so you don't want to have to copy the pg_xlog segments
twice over your network.

No, I don't matter because of the same behavior as 9.1.
Please see the part of "Before:" of the following answer.

What Heikki is proposing will work both when you aren't using
pg_basebackup (as long the output of pg_stop_backup() is somehow
captured in a way that it can be read) and will also work with
pg_basebackup when '-x' isn't specified.

I receive this mail, so I notice I do wrong recognition to what
Heikki is proposing.

my recognition:
Before:
* I thought Heikki proposes, "Execute SQL(pg_start_backup('x'); copy
the data directory and pg_stop_backup();) from the standby server
to the primary server".
-> I disliked this.
Now:
* Heikki is proposing both No using pg_basebackup and Not specify -x.
So,
* Use the output of pg_stop_backup().
* Don't use backup history file.
he thinks.

Right?

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

On 11-06-23 02:41 AM, Jun Ishiduka wrote:

I receive this mail, so I notice I do wrong recognition to what
Heikki is proposing.

my recognition:
Before:
* I thought Heikki proposes, "Execute SQL(pg_start_backup('x'); copy
the data directory and pg_stop_backup();) from the standby server
to the primary server".
-> I disliked this.
Now:
* Heikki is proposing both No using pg_basebackup and Not specify -x.
So,
* Use the output of pg_stop_backup().
* Don't use backup history file.
he thinks.

Right?

What I think he is proposing would not require using pg_stop_backup()
but you could also modify pg_stop_back() to work as well.

What do you think of this idea?

Do you see how the patch can be reworked to accomplish this?

What I think he is proposing would not require using pg_stop_backup()
but you could also modify pg_stop_back() to work as well.

What do you think of this idea?

Do you see how the patch can be reworked to accomplish this?

The logic that not use pg_stop_backup() would be difficult,
because pg_stop_backup() is used to identify minRecoveryPoint.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

On 11-06-24 12:41 AM, Jun Ishiduka wrote:

The logic that not use pg_stop_backup() would be difficult,
because pg_stop_backup() is used to identify minRecoveryPoint.

Considering everything that has been discussed on this thread so far.

Do you still think your patch is the best way to accomplish base backups
from standby servers?
If not what changes do you think should be made?

Steve

Considering everything that has been discussed on this thread so far.

Do you still think your patch is the best way to accomplish base backups
from standby servers?
If not what changes do you think should be made?

I reconsider the way to not use pg_stop_backup().

Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*

Behavior while restore:
* read "Minimum recovery ending location" of the copied pg_control.
* use the value with the same purposes as the end-of-backup location.
-> When the value is equal to 0/0, this behavior do not do.
This situation is to acquire backup from master server.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

On 11-06-28 01:52 AM, Jun Ishiduka wrote:

Considering everything that has been discussed on this thread so far.

Do you still think your patch is the best way to accomplish base backups
from standby servers?
If not what changes do you think should be made?

I reconsider the way to not use pg_stop_backup().

Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*

Behavior while restore:
* read "Minimum recovery ending location" of the copied pg_control.
* use the value with the same purposes as the end-of-backup location.
-> When the value is equal to 0/0, this behavior do not do.
This situation is to acquire backup from master server.

The behaviour you describe above sounds okay to me, if someone else sees
issues with this then they should speak up (ideally before you go off
and write a new patch)

I'm going to consolidate my other comments below so this can act as a
more complete review.

Usability Review
-----------------
We would like to be able to perform base backups from slaves without
having to call pg_start_backup() on the master. We can not currently do
this. The patch tries to do this. From a useability point of view it
would be nice if this could be done both manually with pg_start_backup()
and with pg_basebackup.

The main issue I have with the first patch you submitted is that it does
not work for cases where you don't want to call pg_basebackup or you
don't want to include the wal segments in the output of pg_basebackup.
There are a number of these use-cases (examples include the wal is
already available on an archive server, or you want to use
filesystem/disk array level snapshots instead of tar) . I feel your
above proposal to copy the control file as the last step in the
basebackup and the get the minRecoveryEnding location from this solves
these issues. It would be nice if things 'worked' when calling
pg_basebackup against the slave (maybe by having perform_base_backup()
resend the control file after it has sent the log?).

Feature test & Performance review
-----------------
Skipped since a new patch is coming

Coding Review
------------------
I didn't look too closely at the code since a new patch that might
change a lot of the code. I did like how you added comments to most of
the larger code blocks that you added.

Architecture Review
-----------------------
There were some concerns with your original approach but using the
control file was suggested by Heikki and seems sound to me.

I'm marking this 'waiting for author' , if you don't think you'll be
able to get a reworked patch out during this commitfest then you should
move it to 'returned with feedback'

Steve

2011/6/28 Jun Ishiduka <ishizuka.jun@po.ntts.co.jp>:

Considering everything that has been discussed on this thread so far.

Do you still think your patch is the best way to accomplish base backups
from standby servers?
If not what changes do you think should be made?

I reconsider the way to not use pg_stop_backup().

Process of online base backup on standby server:
 1. pg_start_backup('x');
 2. copy the data directory
 3. copy *pg_control*

Who deletes the backup_label file created by pg_start_backup()?
Isn't pg_stop_backup() required to do that?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Jun 30, 2011 5:59 AM, "Fujii Masao" <masao.fujii@gmail.com> wrote:

2011/6/28 Jun Ishiduka <ishizuka.jun@po.ntts.co.jp>:

Considering everything that has been discussed on this thread so far.

Do you still think your patch is the best way to accomplish base

backups

from standby servers?
If not what changes do you think should be made?

I reconsider the way to not use pg_stop_backup().

Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*

Who deletes the backup_label file created by pg_start_backup()?
Isn't pg_stop_backup() required to do that?

You need it to take the system out of backup mode as well, don't you?

/Magnus

Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*

Who deletes the backup_label file created by pg_start_backup()?
Isn't pg_stop_backup() required to do that?

You need it to take the system out of backup mode as well, don't you?

Certainly so.

Add to the above process:
4. pg_stop_backup();

But I do not consider a case such as to promote in backup mode yet.
I need to think a little, including it.

On this commitfest, the goal of the patch is to be able to be
recovered using "Minimum recovery ending location" in the control file.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

On this commitfest, the goal of the patch is to be able to be
recovered using "Minimum recovery ending location" in the control file.

Done.

Regards.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

2011/7/1 Jun Ishiduka <ishizuka.jun@po.ntts.co.jp>:

On this commitfest, the goal of the patch is to be able to be
recovered using "Minimum recovery ending location" in the control file.

Done.

When the standby restarts after it crashes during recovery, it always
checks whether recovery has reached the backup end location by
using minRecoveryPoint even though the standby doesn't start from
the backup. This looks odd.

-		XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
+		(XLogRecPtrIsInvalid(ControlFile->backupStartPoint) ||
+		 reachedControlMinRecoveryPoint == true))

The flag 'reachedControlMinRecoveryPoint' is really required? When recovery
reaches minRecoveryPoint, ControlFile->backupStartPoint is reset to zero. So
we can check whether recovery has reached minRecoveryPoint or not by only
doing XLogRecPtrIsInvalid(ControlFile->backupStartPoint). No?

We should check if recovery has reached minRecoveryPoint before calling
CheckRecoveryConsistency() after reading new WAL record? Otherwise,
even if recovery has reached minRecoveryPoint, the standby cannot think
that it's in consistent state until it reads new WAL record.

+						if (XLByteLT(ControlFile->minRecoveryPoint, EndRecPtr))
+							ControlFile->minRecoveryPoint = EndRecPtr;

Why does ControlFile->minRecoveryPoint need to be set to EndRecPtr?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

When the standby restarts after it crashes during recovery, it always
checks whether recovery has reached the backup end location by
using minRecoveryPoint even though the standby doesn't start from
the backup. This looks odd.

Certainly.

But, in this case, the state before recovery starts is lost.
Therefore, postgres can not see that the backup got from whether
standby server or master.

What should?
Should use pg_control?
Ex.
* Add 'Where to get backup' to pg_control. (default 'none')
* When recovery starts, it checks it whether 'none'.
* When minRecoveryPoint equals 0/0, change 'master'.
* When minRecoveryPoint do not equals 0/0, change 'slave'.
* When it reached the end of recovery, change 'none' .

-		XLogRecPtrIsInvalid(ControlFile->backupStartPoint))
+		(XLogRecPtrIsInvalid(ControlFile->backupStartPoint) ||
+		 reachedControlMinRecoveryPoint == true))

The flag 'reachedControlMinRecoveryPoint' is really required? When recovery
reaches minRecoveryPoint, ControlFile->backupStartPoint is reset to zero. So
we can check whether recovery has reached minRecoveryPoint or not by only
doing XLogRecPtrIsInvalid(ControlFile->backupStartPoint). No?

Yes.
'reachedControlMinRecoveryPoint' is unnecessary.

We should check if recovery has reached minRecoveryPoint before calling
CheckRecoveryConsistency() after reading new WAL record? Otherwise,
even if recovery has reached minRecoveryPoint, the standby cannot think
that it's in consistent state until it reads new WAL record.

This is a same sequence with a case of backup end location.
It should be no changed.

+						if (XLByteLT(ControlFile->minRecoveryPoint, EndRecPtr))
+							ControlFile->minRecoveryPoint = EndRecPtr;

Why does ControlFile->minRecoveryPoint need to be set to EndRecPtr?

Yes.
I delete it.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

2011/7/4 Jun Ishiduka <ishizuka.jun@po.ntts.co.jp>:

When the standby restarts after it crashes during recovery, it always
checks whether recovery has reached the backup end location by
using minRecoveryPoint even though the standby doesn't start from
the backup. This looks odd.

Certainly.

But, in this case, the state before recovery starts is lost.
Therefore, postgres can not see that the backup got from whether
standby server or master.

What should?
Should use pg_control?
 Ex.
  * Add 'Where to get backup' to pg_control. (default 'none')
  * When recovery starts, it checks it whether 'none'.
     * When minRecoveryPoint equals 0/0, change 'master'.
     * When minRecoveryPoint do not equals 0/0, change 'slave'.
  * When it reached the end of recovery, change 'none' .

What about using backupStartPoint to check whether this recovery
started from the backup or not?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

What about using backupStartPoint to check whether this recovery
started from the backup or not?

No, postgres can check whether this recovery started from the backup
or not, but can not check whether standby server or master (got backup
from).

Once recovery started, backupStartPoint is recorded to pg_control until
recovery reaches backup end location, it is not related to any backup
server.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------

2011/7/5 Jun Ishiduka <ishizuka.jun@po.ntts.co.jp>:

What about using backupStartPoint to check whether this recovery
started from the backup or not?

No, postgres can check whether this recovery started from the backup
or not, but can not check whether standby server or master (got backup
from).

Oh, right. We cannot distinguish the following two cases just by using
minRecoveryPoint and backupStartPoint.

* The standby starts from the backup taken from the standby
* The standby starts after it crashes during recovering from the
backup taken from the master

As you proposed, adding new field which stores the backup end location
taken from minRecoveryPoint, into pg_control sounds good idea.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

As you proposed, adding new field which stores the backup end location
taken from minRecoveryPoint, into pg_control sounds good idea.

Update patch.

Regards.

--------------------------------------------
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka.jun@po.ntts.co.jp
--------------------------------------------