[PATCH] Make pg_basebackup configure and start standby [Review]

Started by Amit Kapilaover 13 years ago58 messageshackers
Jump to latest
#1Amit Kapila
amit.kapila16@gmail.com

On Sun, 01 Jul 2012 13:02:17 +0200 Boszormenyi Zoltan wrote:

attached is a patch that does $SUBJECT.

It's a usability enhancement, to take a backup, write

a minimalistic recovery.conf and start the streaming

standby in one go.

Comments?

[Review of Patch]

Basic stuff:
----------------------
- Patch applies OK
- Compiles cleanly with no warnings

What it does:
-------------------------
The pg_basebackup tool does the backup of Cluster from server to the
specified location.
This new functionality will also writes the recovery.conf in the database
directory and start the standby server based on options passed to
pg_basebackup.

Usability
----------------
For usability aspect, I am not very sure how many users would like to start
the standby server using basebackup.
According to me it can be useful for users who have automated scripts to
start server after backup can use this feature.

Feature Testing:
-----------------------------

1. Test pg_basebackup with option -R to check that the recovery.conf file is
written to data directory.
--recovery.conf file is created in data directory.

2. Test pg_basebackup with option -R to check that the recovery.conf file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.

3. Test pg_basebackup with option -S to check the standby server start on
the same/different machine.
--Starting standby server is success in if pg_basebackup is taken in
different machine.

4. Test pg_basebackup with both options -S and -R to check the standby
server start on same/different machine.
--Starting standby server is success in if pg_basebackup is taken in
different machine.

5. Test pg_basebackup with option -S including -h, -U, -p, -w and -W to
check the standy server start
and verify the recovery.conf which is created in data directory.
--Except password, rest of the primary connection info parameters are
working fine.

6. Test pg_basebackup with conflict options (-x or -X and -R or -S).
--Error is given when the conflict options are provided to
pg_basebackup.

7. Test pg_basebackup with option -S where pg_ctl/postmaster binaries are
not present in the path.
--Error is given as not able to execute.

8. Test pg_basebackup with option -S by connecting to a standby server.
--standby server is started successfully when pg_basebackup is made from
a standby server also.

Code Review:
----------------------------
1. In function WriteRecoveryConf(), un-initialized filename is used.
due to which it can print junk for below line in code
printf("add password to primary_conninfo in %s if needed\n", filename);

2. In function WriteRecoveryConf(), in below code if fopen fails (due to
disk full or any other file related error) it will print the error and
exits.
So now it can be confusing to user, in respect to can he consider
backup as successfull and proceed. IMO, either error meesage or
documentation
can suggest the for such error user can proceed with backup to write
his own recovery.conf and start the standby.

+        cf = fopen(filename, "w"); 
+        if (cf == NULL) 
+        { 
+                fprintf(stderr, _("cannot create %s"), filename); 
+                exit(1); 
+        } 

3. In function main,
instead of the following code it can be changed in two different ways,

if (startstandby)
writerecoveryconf = true;

change1:
case 'R':
writerecoveryconf = true;
break;
case 'S':
startstandby = true;
writerecoveryconf = true;
break;

change2:
case 'S':
startstandby = true;
case 'R':
writerecoveryconf = true;
break;

4. The password is not written to primary_conninfo even if the dbpassword is
present because of this reason
connecting to the primary is failing because of authentication failure.

5. write the function header for the newly added functions.

6. execvp function is deprecated beginning in Visual C++ 2005. which is used
to fork the pg_ctl process.
http://msdn.microsoft.com/en-us/library/ms235414.aspx

7. In StartStandby function, it is better to free the memory allocated for
path (path = xstrdup(command);)

Defects:
-------------
1. If the pg_basebackup is used in the same machine with the option of -S,
the standby server start
will fail as the port already in use because of using the same
postgresql.conf.

2. If the hot_standby=off in master conf file, the same is copied to
subscriber and starts the server. with that
no client connections are allowed to the server.

Documentation issues:
--------------------------------
1. For -R option,
Conflicts with <option>--xlog
I think it is better to explain the reason of conflict.

2. For -S option,
"Start the standby database server. Implies -R option."
I think the above can be improved to
"Writes the recovery.conf and start the standby database server. There
is no need for user to specify -R option explicitly."
or something similar.

With Regards,

Amit Kapila.

#2Boszormenyi Zoltan
zb@cybertec.at
In reply to: Amit Kapila (#1)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

Hi,

first, thanks for the review. Comments are below.

2012-09-20 12:30 keltezéssel, Amit Kapila írta:

On Sun, 01 Jul 2012 13:02:17 +0200 Boszormenyi Zoltan wrote:

attached is a patch that does $SUBJECT.

It's a usability enhancement, to take a backup, write

a minimalistic recovery.conf and start the streaming

standby in one go.

Comments?

*[Review of Patch]*

*Basic stuff:*
----------------------
- Patch applies OK

This is not true anymore with a newer GIT version.
Some chunk for pg_basebackup.c was rejected.

- Compiles cleanly with no warnings

*What it does:*
-------------------------
The pg_basebackup tool does the backup of Cluster from server to the specified location.
This new functionality will also writes the recovery.conf in the database directory and
start the standby server based on options passed to pg_basebackup.

*Usability*
*----------------*
For usability aspect, I am not very sure how many users would like to start the standby
server using basebackup.

Also, Magnus raised the point that it wouldn't really work on MS Windows
where you *have to* start the service via OS facilities. This part of the patch
was killed.

According to me it can be useful for users who have automated scripts to start server
after backup can use this feature.

Well, scripting is not portable across UNIXes and Windows,
you have to spell out starting the server differently.

*Feature Testing:*
-----------------------------

1. Test pg_basebackup with option -R to check that the recovery.conf file is written to
data directory.
--recovery.conf file is created in data directory.

2. Test pg_basebackup with option -R to check that the recovery.conf file is not able to
create because of disk full.
--Error is given as recovery.conf file is not able to create.

3. Test pg_basebackup with option -S to check the standby server start on the
same/different machine.
--Starting standby server is success in if pg_basebackup is taken in different machine.

4. Test pg_basebackup with both options -S and -R to check the standby server start on
same/different machine.
--Starting standby server is success in if pg_basebackup is taken in different machine.

5. Test pg_basebackup with option -S including -h, -U, -p, -w and -W to check the standy
server start
and verify the recovery.conf which is created in data directory.
--Except password, rest of the primary connection info parameters are working fine.

The password part is now fixed.

6. Test pg_basebackup with conflict options (-x or -X and -R or -S).
--Error is given when the conflict options are provided to pg_basebackup.

7. Test pg_basebackup with option -S where pg_ctl/postmaster binaries are not present in
the path.
--Error is given as not able to execute.

8. Test pg_basebackup with option -S by connecting to a standby server.
--standby server is started successfully when pg_basebackup is made from a standby
server also.

*Code Review:*
----------------------------
1. In function WriteRecoveryConf(), un-initialized filename is used.
due to which it can print junk for below line in code
printf("add password to primary_conninfo in %s if needed\n", filename);

Fixed.

2. In function WriteRecoveryConf(), in below code if fopen fails (due to disk full or
any other file related error) it will print the error and exits.
So now it can be confusing to user, in respect to can he consider backup as
successfull and proceed. IMO, either error meesage or documentation
can suggest the for such error user can proceed with backup to write his own
recovery.conf and start the standby.

+        cf = fopen(filename, "w");
+        if (cf == NULL)
+        {
+                fprintf(stderr, _("cannot create %s"), filename);
+                exit(1);
+        }

But BaseBackup() function did indicate already that it has
finished successfully with

if (verbose)
fprintf(stderr, "%s: base backup completed\n", progname);

Would it be an expected (as in: not confusing) behaviour to return 0
from pg_basebackup if the backup itself has finished, but failed to write
the recovery.conf or start the standby if those were requested?

I have modified my WriteRecoveryConf() to do exit(2) instead of exit(1)
to indicate a different error. exit(1) seems to be for reporting configuration
or connection errors. (I may be mistaken though.)

3. In function main,
instead of the following code it can be changed in two different ways,

if (startstandby)
writerecoveryconf = true;

change1:
case 'R':
writerecoveryconf = true;
break;
case 'S':
startstandby = true;
writerecoveryconf = true;
break;

change2:
case 'S':
startstandby = true;
case 'R':
writerecoveryconf = true;
break;

I went with your second variant at first but it's not needed anymore
as only "-R" exists.

4. The password is not written to primary_conninfo even if the dbpassword is present
because of this reason
connecting to the primary is failing because of authentication failure.

Fixed.

5. write the function header for the newly added functions.

Fixed.

6. execvp function is deprecated beginning in Visual C++ 2005. which is used to fork the
pg_ctl process.
http://msdn.microsoft.com/en-us/library/ms235414.aspx

This issue is now irrelevant as the standby is not started, there is no "-S" option.

7. In StartStandby function, it is better to free the memory allocated for path (path =
xstrdup(command);)

Same as above.

*Defects:*
*-------------*
1. If the pg_basebackup is used in the same machine with the option of -S, the standby
server start
will fail as the port already in use because of using the same postgresql.conf.

Well, running initdb twice on the same machine with different data directories
would also cause the second server fail to start because of the same issue
and it's not called a bug. I think this is irrelevant as is and also because there
is no "-S" now.

2. If the hot_standby=off in master conf file, the same is copied to subscriber and
starts the server. with that
no client connections are allowed to the server.

Well, it simply copies the source server behaviour, which can also be a
replication standby. PostgreSQL has cascading replication, you know.

*Documentation issues:*
*--------------------------------*
1. For -R option,
Conflicts with <option>--xlog
I think it is better to explain the reason of conflict.

Fixed.

2. For -S option,
"Start the standby database server. Implies -R option."
I think the above can be improved to
"Writes the recovery.conf and start the standby database server. There is no need for
user to specify -R option explicitly."
or something similar.

Not relevant anymore.

Again, thanks for the review.

The second generation of this work is now attached and contains a new
feature as was discussed and suggested by Magnus Hagander, Fujii Masao
and Peter Eisentraut. So libpq has grown a new function:

+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn);

This copies all the connection parameters back from the live PGconn itself
so everything that's needed to connect is already validated.

This is used by the second patch which makes the changes in pg_basebackup
simpler and not hardcoded.

Please, review.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

02-pg_basebackup-v2.patchtext/x-patch; name=02-pg_basebackup-v2.patchDownload+107-1
01-PQconninfo-v3.patchtext/x-patch; name=01-PQconninfo-v3.patchDownload+87-0
#3Peter Eisentraut
peter_e@gmx.net
In reply to: Boszormenyi Zoltan (#2)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Wed, 2012-10-03 at 18:15 +0200, Boszormenyi Zoltan wrote:

The second generation of this work is now attached and contains a new
feature as was discussed and suggested by Magnus Hagander, Fujii Masao
and Peter Eisentraut. So libpq has grown a new function:

+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn);

This copies all the connection parameters back from the live PGconn
itself
so everything that's needed to connect is already validated.

I don't like that this code maintains a second list of all possible
libpq connection parameters. The parameters to add to the connection
string should be driven off the authoritative list in PQconninfoOptions.

#4Boszormenyi Zoltan
zb@cybertec.at
In reply to: Peter Eisentraut (#3)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-04 05:24 keltezéssel, Peter Eisentraut írta:

On Wed, 2012-10-03 at 18:15 +0200, Boszormenyi Zoltan wrote:

The second generation of this work is now attached and contains a new
feature as was discussed and suggested by Magnus Hagander, Fujii Masao
and Peter Eisentraut. So libpq has grown a new function:

+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn);

This copies all the connection parameters back from the live PGconn
itself
so everything that's needed to connect is already validated.

I don't like that this code maintains a second list of all possible
libpq connection parameters.

Where does it do that? In PQconninfo() itself? Why is it a problem?
Or to put it bluntly: the same problem is in fillPGconn() too, as it also
has the same set of parameters listed. So there is already code
that you don't like. :-)

How about a static mapping between option names and
offsetof(struct pg_conn, member) values? This way both fillPGconn()
and PQconninfo() can avoid maintaining the list of parameter names.

The parameters to add to the connection
string should be driven off the authoritative list in PQconninfoOptions.

So, should I add a second flag to PQconninfoOption to indicate that
certain options should not be used for primary_conninfo?

Thanks and best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

#5Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#4)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-04 06:47 keltezéssel, Boszormenyi Zoltan írta:

2012-10-04 05:24 keltezéssel, Peter Eisentraut írta:

On Wed, 2012-10-03 at 18:15 +0200, Boszormenyi Zoltan wrote:

The second generation of this work is now attached and contains a new
feature as was discussed and suggested by Magnus Hagander, Fujii Masao
and Peter Eisentraut. So libpq has grown a new function:

+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn);

This copies all the connection parameters back from the live PGconn
itself
so everything that's needed to connect is already validated.

I don't like that this code maintains a second list of all possible
libpq connection parameters.

Where does it do that? In PQconninfo() itself? Why is it a problem?
Or to put it bluntly: the same problem is in fillPGconn() too, as it also
has the same set of parameters listed. So there is already code
that you don't like. :-)

How about a static mapping between option names and
offsetof(struct pg_conn, member) values? This way both fillPGconn()
and PQconninfo() can avoid maintaining the list of parameter names.

Did you think about something like the attached code?

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v7.patchtext/x-patch; name=01-PQconninfo-v7.patchDownload+187-94
02-pg_basebackup-v5.patchtext/x-patch; name=02-pg_basebackup-v5.patchDownload+111-1
#6Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#5)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-04 12:42 keltezéssel, Boszormenyi Zoltan írta:

2012-10-04 06:47 keltezéssel, Boszormenyi Zoltan írta:

2012-10-04 05:24 keltezéssel, Peter Eisentraut írta:

On Wed, 2012-10-03 at 18:15 +0200, Boszormenyi Zoltan wrote:

The second generation of this work is now attached and contains a new
feature as was discussed and suggested by Magnus Hagander, Fujii Masao
and Peter Eisentraut. So libpq has grown a new function:

+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn);

This copies all the connection parameters back from the live PGconn
itself
so everything that's needed to connect is already validated.

I don't like that this code maintains a second list of all possible
libpq connection parameters.

Where does it do that? In PQconninfo() itself? Why is it a problem?
Or to put it bluntly: the same problem is in fillPGconn() too, as it also
has the same set of parameters listed. So there is already code
that you don't like. :-)

How about a static mapping between option names and
offsetof(struct pg_conn, member) values? This way both fillPGconn()
and PQconninfo() can avoid maintaining the list of parameter names.

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and PQconninfo() are
symmetric and work for "requiressl".

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v8.patchtext/x-patch; name=01-PQconninfo-v8.patchDownload+194-94
02-pg_basebackup-v5.patchtext/x-patch; name=02-pg_basebackup-v5.patchDownload+111-1
#7Tom Lane
tgl@sss.pgh.pa.us
In reply to: Boszormenyi Zoltan (#6)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and PQconninfo() are
symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need
to put that info somewhere else.

regards, tom lane

#8Boszormenyi Zoltan
zb@cybertec.at
In reply to: Tom Lane (#7)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and PQconninfo() are
symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need
to put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden
in the PQconninfoMapping[] array and PQconninfo(conn, true)
pre-filters the list as in the attached version.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v9.patchtext/x-patch; name=01-PQconninfo-v9.patchDownload+206-71
02-pg_basebackup-v6.patchtext/x-patch; name=02-pg_basebackup-v6.patchDownload+111-1
#9Amit Kapila
amit.kapila16@gmail.com
In reply to: Boszormenyi Zoltan (#8)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Thursday, October 04, 2012 9:50 PM Boszormenyi Zoltan

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and
PQconninfo() are symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need to
put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden in the
PQconninfoMapping[] array and PQconninfo(conn, true) pre-filters the
list as in the attached version.

Please find the review of the patch.

Basic stuff:
------------
- patch apply failed at exports.txt file. Needs rebase to the current
master.
- Compiles cleanly with no warnings

What it does:
--------------
pg_basebackup does the base backup from the primary machine and also writes
the recovery.conf file with the option -R,
which is used for the standby to connect to primary for streaming
replication.

Testing:
---------
1. Test pg_basebackup with option -R to check that the recovery.conf file is
written to data directory.
--recovery.conf file is created in data directory.

2. Test pg_basebackup with option -R to check that the recovery.conf file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.

3. Test pg_basebackup with option -R including -h, -U, -p, -w and -W.
verify the recovery.conf which is created in data directory.
--All the primary connection info parameters are working fine.

4. Test pg_basebackup with conflict options (-x or -X and -R).
--Error is given when the conflict options are provided to
pg_basebackup.

5. Test pg_basebackup with option -R from a standby server.
--recovery.conf file is created in data directory.

Code Review: 
------------- 
1. 
typedef struct PQconninfoMapping { 
+        char                *keyword; 
+        size_t                member_offset; 
+        bool                for_replication; 
+        char                *conninfoValue;        /* Special value mapping
*/ 
+        char                *connValue; 
+} PQconninfoMapping; 

Provide the better explanation of conninfoValue and connValue, how and where
these are used?

2. if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)

In any case if the above condition is not satisfied then the PGconn data is
not filled with PQconninfoOption.
Is it correct?

Documentation: 
------------- 
1. 
+        <para> 
+       The <literal>for_replication</> argument can be used to exclude some
+       options from the list which are used by the walreceiver module. 
+       <application>pg_basebackup</application> uses this pre-filtered list
+       to construct <literal>primary_conninfo</> in the automatically
generated 
+       recovery.conf file. 
+      </para> 

I feel the explanation should be as follows,
exclude some options from the list which are not used by the walreceiver
module?

Observations/Issues:
-------------------
1. If the password contains any escape sequence characters, It is leading to
problems while walreceiver connecting to primary
by using the primary conninfo from recovery.conf

please log an warning message or a note in document to handle such a case
manually by the user.

With Regards,
Amit Kapila.

#10Boszormenyi Zoltan
zb@cybertec.at
In reply to: Amit Kapila (#9)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

Hi,

thanks for the new review.

2012-10-10 08:58 keltezéssel, Amit Kapila írta:

On Thursday, October 04, 2012 9:50 PM Boszormenyi Zoltan

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and
PQconninfo() are symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need to
put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden in the
PQconninfoMapping[] array and PQconninfo(conn, true) pre-filters the
list as in the attached version.

Please find the review of the patch.

Basic stuff:
------------
- patch apply failed at exports.txt file. Needs rebase to the current
master.

Done.

- Compiles cleanly with no warnings

What it does:
--------------
pg_basebackup does the base backup from the primary machine and also writes
the recovery.conf file with the option -R,
which is used for the standby to connect to primary for streaming
replication.

Testing:
---------
1. Test pg_basebackup with option -R to check that the recovery.conf file is
written to data directory.
--recovery.conf file is created in data directory.

2. Test pg_basebackup with option -R to check that the recovery.conf file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.

3. Test pg_basebackup with option -R including -h, -U, -p, -w and -W.
verify the recovery.conf which is created in data directory.
--All the primary connection info parameters are working fine.

4. Test pg_basebackup with conflict options (-x or -X and -R).
--Error is given when the conflict options are provided to
pg_basebackup.

5. Test pg_basebackup with option -R from a standby server.
--recovery.conf file is created in data directory.

Code Review:
-------------
1.
typedef struct PQconninfoMapping {
+        char                *keyword;
+        size_t                member_offset;
+        bool                for_replication;
+        char                *conninfoValue;        /* Special value mapping
*/
+        char                *connValue;
+} PQconninfoMapping;

Provide the better explanation of conninfoValue and connValue, how and where
these are used?

OK. This is only used for " requiressl='1' " (in the connection string)
and if '1' is set, PGconn.sslmode will be set to "require". All other
values are treated as it's being unset. This simplistic mapping
is used because there is only one such setting where different values
are used on the conninfo and the PGconn sides.

2. if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)

In any case if the above condition is not satisfied then the PGconn data is
not filled with PQconninfoOption.
Is it correct?

Yes, it stays NULL as makeEmptyPGconn() initialized it. This case only happens
with the "requiressl" setting with its special mapping. If you set " requiressl = '0' "
then it means that " requiressl='1' " was not set so the PGconn side stays as NULL.

The special casing was there in the old code too and behaves the same.

Documentation:
-------------
1.
+        <para>
+       The <literal>for_replication</> argument can be used to exclude some
+       options from the list which are used by the walreceiver module.
+       <application>pg_basebackup</application> uses this pre-filtered list
+       to construct <literal>primary_conninfo</> in the automatically
generated
+       recovery.conf file.
+      </para>

I feel the explanation should be as follows,
exclude some options from the list which are not used by the walreceiver
module?

Err, no. The list excludes those[1]Actually, more than these 3 options are excluded. The deprecated ones are also excluded. that *are* used (would be
overridden) by the walreceiver module:

----8<--------8<--------8<--------8<--------8<----
static bool
libpqrcv_connect(char *conninfo, XLogRecPtr startpoint)
{
...
snprintf(conninfo_repl, sizeof(conninfo_repl),
"%s dbname=replication replication=true
fallback_application_name=walreceiver",
conninfo);
----8<--------8<--------8<--------8<--------8<----

[1]: Actually, more than these 3 options are excluded. The deprecated ones are also excluded.
ones are also excluded.

Observations/Issues:
-------------------
1. If the password contains any escape sequence characters, It is leading to
problems while walreceiver connecting to primary
by using the primary conninfo from recovery.conf

please log an warning message or a note in document to handle such a case
manually by the user.

Done at both places.

Also, to adapt to the style of other error messages, now all my fprintf() statements
are prefixed with: "%s: ...", progname.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v11.patchtext/x-patch; name=01-PQconninfo-v11.patchDownload+210-71
02-pg_basebackup-v8.patchtext/x-patch; name=02-pg_basebackup-v8.patchDownload+122-1
#11Fujii Masao
masao.fujii@gmail.com
In reply to: Boszormenyi Zoltan (#10)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Wed, Oct 10, 2012 at 10:12 PM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

Hi,

thanks for the new review.

2012-10-10 08:58 keltezéssel, Amit Kapila írta:

On Thursday, October 04, 2012 9:50 PM Boszormenyi Zoltan

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and
PQconninfo() are symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need to
put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden in the
PQconninfoMapping[] array and PQconninfo(conn, true) pre-filters the
list as in the attached version.

Please find the review of the patch.

Basic stuff:
------------
- patch apply failed at exports.txt file. Needs rebase to the current
master.

Done.

- Compiles cleanly with no warnings

What it does:
--------------
pg_basebackup does the base backup from the primary machine and also
writes
the recovery.conf file with the option -R,
which is used for the standby to connect to primary for streaming
replication.

Testing:
---------
1. Test pg_basebackup with option -R to check that the recovery.conf file
is
written to data directory.
--recovery.conf file is created in data directory.
2. Test pg_basebackup with option -R to check that the recovery.conf
file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.
3. Test pg_basebackup with option -R including -h, -U, -p, -w and -W.

verify the recovery.conf which is created in data directory.
--All the primary connection info parameters are working fine.
4. Test pg_basebackup with conflict options (-x or -X and -R).

--Error is given when the conflict options are provided to
pg_basebackup.

5. Test pg_basebackup with option -R from a standby server.
--recovery.conf file is created in data directory.

Code Review:
-------------
1.
typedef struct PQconninfoMapping {
+        char                *keyword;
+        size_t                member_offset;
+        bool                for_replication;
+        char                *conninfoValue;        /* Special value
mapping
*/
+        char                *connValue;
+} PQconninfoMapping;

Provide the better explanation of conninfoValue and connValue, how and
where
these are used?

OK. This is only used for " requiressl='1' " (in the connection string)
and if '1' is set, PGconn.sslmode will be set to "require". All other
values are treated as it's being unset. This simplistic mapping
is used because there is only one such setting where different values
are used on the conninfo and the PGconn sides.

2. if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)

In any case if the above condition is not satisfied then the PGconn data
is
not filled with PQconninfoOption.
Is it correct?

Yes, it stays NULL as makeEmptyPGconn() initialized it. This case only
happens
with the "requiressl" setting with its special mapping. If you set "
requiressl = '0' "
then it means that " requiressl='1' " was not set so the PGconn side stays
as NULL.

The special casing was there in the old code too and behaves the same.

Documentation:
-------------
1.
+        <para>
+       The <literal>for_replication</> argument can be used to exclude
some
+       options from the list which are used by the walreceiver module.
+       <application>pg_basebackup</application> uses this pre-filtered
list
+       to construct <literal>primary_conninfo</> in the automatically
generated
+       recovery.conf file.
+      </para>

I feel the explanation should be as follows,
exclude some options from the list which are not used by the walreceiver
module?

Err, no. The list excludes those[1] that *are* used (would be
overridden) by the walreceiver module:

----8<--------8<--------8<--------8<--------8<----
static bool
libpqrcv_connect(char *conninfo, XLogRecPtr startpoint)
{
...
snprintf(conninfo_repl, sizeof(conninfo_repl),
"%s dbname=replication replication=true
fallback_application_name=walreceiver",
conninfo);
----8<--------8<--------8<--------8<--------8<----

[1] Actually, more than these 3 options are excluded. The deprecated
ones are also excluded.

Observations/Issues:
-------------------
1. If the password contains any escape sequence characters, It is leading
to
problems while walreceiver connecting to primary
by using the primary conninfo from recovery.conf

please log an warning message or a note in document to handle such a
case
manually by the user.

Done at both places.

Also, to adapt to the style of other error messages, now all my fprintf()
statements
are prefixed with: "%s: ...", progname.

In new patches, when I ran "pg_basebackup -D hoge -c fast -R" on MacOS,
I got the following error message. BTW, I compiled the patched PostgreSQL
with --enable-debug and --enable-cassert options.

pg_basebackup(41751) malloc: *** error for object 0x106001af0: pointer
being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

$ uname -a
Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23
16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64

When tar output format is specified together with -R option, recovery.conf is
not included in base.tar. I think it should.

+        setting up the standby. Since creating a backup for a standalone
+        server with <option>-x</option> or <option>-X</option> and adding
+        a recovery.conf to it wouldn't make a working standby, these options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

Regards,

--
Fujii Masao

#12Boszormenyi Zoltan
zb@cybertec.at
In reply to: Fujii Masao (#11)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

On Wed, Oct 10, 2012 at 10:12 PM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

Hi,

thanks for the new review.

2012-10-10 08:58 keltezéssel, Amit Kapila írta:

On Thursday, October 04, 2012 9:50 PM Boszormenyi Zoltan

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and
PQconninfo() are symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need to
put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden in the
PQconninfoMapping[] array and PQconninfo(conn, true) pre-filters the
list as in the attached version.

Please find the review of the patch.

Basic stuff:
------------
- patch apply failed at exports.txt file. Needs rebase to the current
master.

Done.

- Compiles cleanly with no warnings

What it does:
--------------
pg_basebackup does the base backup from the primary machine and also
writes
the recovery.conf file with the option -R,
which is used for the standby to connect to primary for streaming
replication.

Testing:
---------
1. Test pg_basebackup with option -R to check that the recovery.conf file
is
written to data directory.
--recovery.conf file is created in data directory.
2. Test pg_basebackup with option -R to check that the recovery.conf
file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.
3. Test pg_basebackup with option -R including -h, -U, -p, -w and -W.

verify the recovery.conf which is created in data directory.
--All the primary connection info parameters are working fine.
4. Test pg_basebackup with conflict options (-x or -X and -R).

--Error is given when the conflict options are provided to
pg_basebackup.

5. Test pg_basebackup with option -R from a standby server.
--recovery.conf file is created in data directory.

Code Review:
-------------
1.
typedef struct PQconninfoMapping {
+        char                *keyword;
+        size_t                member_offset;
+        bool                for_replication;
+        char                *conninfoValue;        /* Special value
mapping
*/
+        char                *connValue;
+} PQconninfoMapping;

Provide the better explanation of conninfoValue and connValue, how and
where
these are used?

OK. This is only used for " requiressl='1' " (in the connection string)
and if '1' is set, PGconn.sslmode will be set to "require". All other
values are treated as it's being unset. This simplistic mapping
is used because there is only one such setting where different values
are used on the conninfo and the PGconn sides.

2. if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)

In any case if the above condition is not satisfied then the PGconn data
is
not filled with PQconninfoOption.
Is it correct?

Yes, it stays NULL as makeEmptyPGconn() initialized it. This case only
happens
with the "requiressl" setting with its special mapping. If you set "
requiressl = '0' "
then it means that " requiressl='1' " was not set so the PGconn side stays
as NULL.

The special casing was there in the old code too and behaves the same.

Documentation:
-------------
1.
+        <para>
+       The <literal>for_replication</> argument can be used to exclude
some
+       options from the list which are used by the walreceiver module.
+       <application>pg_basebackup</application> uses this pre-filtered
list
+       to construct <literal>primary_conninfo</> in the automatically
generated
+       recovery.conf file.
+      </para>

I feel the explanation should be as follows,
exclude some options from the list which are not used by the walreceiver
module?

Err, no. The list excludes those[1] that *are* used (would be
overridden) by the walreceiver module:

----8<--------8<--------8<--------8<--------8<----
static bool
libpqrcv_connect(char *conninfo, XLogRecPtr startpoint)
{
...
snprintf(conninfo_repl, sizeof(conninfo_repl),
"%s dbname=replication replication=true
fallback_application_name=walreceiver",
conninfo);
----8<--------8<--------8<--------8<--------8<----

[1] Actually, more than these 3 options are excluded. The deprecated
ones are also excluded.

Observations/Issues:
-------------------
1. If the password contains any escape sequence characters, It is leading
to
problems while walreceiver connecting to primary
by using the primary conninfo from recovery.conf

please log an warning message or a note in document to handle such a
case
manually by the user.

Done at both places.

Also, to adapt to the style of other error messages, now all my fprintf()
statements
are prefixed with: "%s: ...", progname.

In new patches, when I ran "pg_basebackup -D hoge -c fast -R" on MacOS,
I got the following error message. BTW, I compiled the patched PostgreSQL
with --enable-debug and --enable-cassert options.

pg_basebackup(41751) malloc: *** error for object 0x106001af0: pointer
being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Can you show a backtrace? I compiled it on Fedora 17/x86_64 with
--enable-depend --enable-debug --enable-cassert. GLIBC is also smart
enough to catch improper free() calls, too.

$ uname -a
Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23
16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64

When tar output format is specified together with -R option, recovery.conf is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

+        setting up the standby. Since creating a backup for a standalone
+        server with <option>-x</option> or <option>-X</option> and adding
+        a recovery.conf to it wouldn't make a working standby, these options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

I don't know. Pointers?

Thanks,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

#13Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#12)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-10 20:36 keltezéssel, Boszormenyi Zoltan írta:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

On Wed, Oct 10, 2012 at 10:12 PM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

Hi,

thanks for the new review.

2012-10-10 08:58 keltezéssel, Amit Kapila írta:

On Thursday, October 04, 2012 9:50 PM Boszormenyi Zoltan

2012-10-04 16:43 keltezéssel, Tom Lane írta:

Boszormenyi Zoltan <zb@cybertec.at> writes:

Did you think about something like the attached code?

Or rather this one, which fixes a bug so fillPGconn() and
PQconninfo() are symmetric and work for "requiressl".

That's incredibly ugly. I'm not sure where we should keep the "R"
information, but shoehorning it into the existing PQconninfoOption
struct like that seems totally unacceptable. Either we're willing to
break backwards compatibility on the Disp-Char strings, or we need to
put that info somewhere else.

I hope only handling the new flag part is ugly. It can be hidden in the
PQconninfoMapping[] array and PQconninfo(conn, true) pre-filters the
list as in the attached version.

Please find the review of the patch.

Basic stuff:
------------
- patch apply failed at exports.txt file. Needs rebase to the current
master.

Done.

- Compiles cleanly with no warnings

What it does:
--------------
pg_basebackup does the base backup from the primary machine and also
writes
the recovery.conf file with the option -R,
which is used for the standby to connect to primary for streaming
replication.

Testing:
---------
1. Test pg_basebackup with option -R to check that the recovery.conf file
is
written to data directory.
--recovery.conf file is created in data directory.
2. Test pg_basebackup with option -R to check that the recovery.conf
file is
not able to create because of disk full.
--Error is given as recovery.conf file is not able to create.
3. Test pg_basebackup with option -R including -h, -U, -p, -w and -W.

verify the recovery.conf which is created in data directory.
--All the primary connection info parameters are working fine.
4. Test pg_basebackup with conflict options (-x or -X and -R).

--Error is given when the conflict options are provided to
pg_basebackup.

5. Test pg_basebackup with option -R from a standby server.
--recovery.conf file is created in data directory.

Code Review:
-------------
1.
typedef struct PQconninfoMapping {
+        char                *keyword;
+        size_t                member_offset;
+        bool                for_replication;
+        char                *conninfoValue;        /* Special value
mapping
*/
+        char                *connValue;
+} PQconninfoMapping;

Provide the better explanation of conninfoValue and connValue, how and
where
these are used?

OK. This is only used for " requiressl='1' " (in the connection string)
and if '1' is set, PGconn.sslmode will be set to "require". All other
values are treated as it's being unset. This simplistic mapping
is used because there is only one such setting where different values
are used on the conninfo and the PGconn sides.

2. if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)

In any case if the above condition is not satisfied then the PGconn data
is
not filled with PQconninfoOption.
Is it correct?

Yes, it stays NULL as makeEmptyPGconn() initialized it. This case only
happens
with the "requiressl" setting with its special mapping. If you set "
requiressl = '0' "
then it means that " requiressl='1' " was not set so the PGconn side stays
as NULL.

The special casing was there in the old code too and behaves the same.

Documentation:
-------------
1.
+        <para>
+       The <literal>for_replication</> argument can be used to exclude
some
+       options from the list which are used by the walreceiver module.
+       <application>pg_basebackup</application> uses this pre-filtered
list
+       to construct <literal>primary_conninfo</> in the automatically
generated
+       recovery.conf file.
+      </para>

I feel the explanation should be as follows,
exclude some options from the list which are not used by the walreceiver
module?

Err, no. The list excludes those[1] that *are* used (would be
overridden) by the walreceiver module:

----8<--------8<--------8<--------8<--------8<----
static bool
libpqrcv_connect(char *conninfo, XLogRecPtr startpoint)
{
...
snprintf(conninfo_repl, sizeof(conninfo_repl),
"%s dbname=replication replication=true
fallback_application_name=walreceiver",
conninfo);
----8<--------8<--------8<--------8<--------8<----

[1] Actually, more than these 3 options are excluded. The deprecated
ones are also excluded.

Observations/Issues:
-------------------
1. If the password contains any escape sequence characters, It is leading
to
problems while walreceiver connecting to primary
by using the primary conninfo from recovery.conf

please log an warning message or a note in document to handle such a
case
manually by the user.

Done at both places.

Also, to adapt to the style of other error messages, now all my fprintf()
statements
are prefixed with: "%s: ...", progname.

In new patches, when I ran "pg_basebackup -D hoge -c fast -R" on MacOS,
I got the following error message. BTW, I compiled the patched PostgreSQL
with --enable-debug and --enable-cassert options.

pg_basebackup(41751) malloc: *** error for object 0x106001af0: pointer
being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

Can you show a backtrace? I compiled it on Fedora 17/x86_64 with
--enable-depend --enable-debug --enable-cassert. GLIBC is also smart
enough to catch improper free() calls, too.

I was able to test it on OSX and I found my bug. Attached is the new code.
The problem was in conninfo_init(), the last entry in the filtered list was
not initialized to 0. It seems that for some reason, my Linux machine gave
me pre-initialized memory.

$ uname -a
Darwin hrk.local 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23
16:25:48 PDT 2012; root:xnu-1699.32.7~1/RELEASE_X86_64 x86_64

When tar output format is specified together with -R option, recovery.conf is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

+        setting up the standby. Since creating a backup for a standalone
+        server with <option>-x</option> or <option>-X</option> and adding
+        a recovery.conf to it wouldn't make a working standby, these options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

I don't know. Pointers?

Thanks,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v12.patchtext/x-patch; name=01-PQconninfo-v12.patchDownload+211-71
02-pg_basebackup-v9.patchtext/x-patch; name=02-pg_basebackup-v9.patchDownload+122-1
#14Fujii Masao
masao.fujii@gmail.com
In reply to: Boszormenyi Zoltan (#12)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Thu, Oct 11, 2012 at 3:36 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

When tar output format is specified together with -R option, recovery.conf
is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

Because it's more user-friendly. If recovery.conf is not included in base.tar,
when base.tar is extracted to disk to use the backup, a user always needs
to copy recovery.conf to the extracted directory. OTOH if it's included in
base.tar, such copy operation is not required and we can simplify the
procedures to use the backup a bit.

+        setting up the standby. Since creating a backup for a standalone
+        server with <option>-x</option> or <option>-X</option> and adding
+        a recovery.conf to it wouldn't make a working standby, these
options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

I don't know. Pointers?

There is no restriction that the backup which was taken by using
pg_basebackup -x cannot be used to start the standby. So I wonder
why -R option cannot work together with -x. It's useful if recovery.conf
is automatically written even with pg_basebackup -x.

And I found another problem: when -(stdout) is specified in -D option,
recovery.conf fails to be written.

$ pg_basebackup -D- -F t -c fast -R > hoge.tar
NOTICE: WAL archiving is not enabled; you must ensure that all
required WAL segments are copied through other means to complete the
backup
pg_basebackup: cannot create -/recovery.conf: No such file or directory

Regards,

--
Fujii Masao

#15Fujii Masao
masao.fujii@gmail.com
In reply to: Boszormenyi Zoltan (#13)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Thu, Oct 11, 2012 at 4:58 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

I was able to test it on OSX and I found my bug. Attached is the new code.
The problem was in conninfo_init(), the last entry in the filtered list was
not initialized to 0. It seems that for some reason, my Linux machine gave
me pre-initialized memory.

Thanks. Will test.

Regards,

--
Fujii Masao

#16Robert Haas
robertmhaas@gmail.com
In reply to: Fujii Masao (#14)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Wed, Oct 10, 2012 at 8:02 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Oct 11, 2012 at 3:36 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

When tar output format is specified together with -R option, recovery.conf
is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

Because it's more user-friendly. If recovery.conf is not included in base.tar,
when base.tar is extracted to disk to use the backup, a user always needs
to copy recovery.conf to the extracted directory. OTOH if it's included in
base.tar, such copy operation is not required and we can simplify the
procedures to use the backup a bit.

+1.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#17Boszormenyi Zoltan
zb@cybertec.at
In reply to: Robert Haas (#16)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-12 22:21 keltezéssel, Robert Haas írta:

On Wed, Oct 10, 2012 at 8:02 PM, Fujii Masao <masao.fujii@gmail.com> wrote:

On Thu, Oct 11, 2012 at 3:36 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

When tar output format is specified together with -R option, recovery.conf
is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

Because it's more user-friendly. If recovery.conf is not included in base.tar,
when base.tar is extracted to disk to use the backup, a user always needs
to copy recovery.conf to the extracted directory. OTOH if it's included in
base.tar, such copy operation is not required and we can simplify the
procedures to use the backup a bit.

+1.

OK, out of popular demand, I implemented writing into the base.tar
if both -R and -Ft was specified.

The code to create a tar header and the checksum inside the header
was copied from bin/pg_dump/pg_backup_tar.c.

I tested these scenarios ("server" can be either a master and standby):
- backup a server with -Fp
(no recovery.conf in the target directory was written)
- backup a server with -Ftar
(no recovery.conf was written into the target directory or base.tar)
- backup a server with -Ftar -z
(no recovery.conf was written into the target directory or base.tar.gz)
- backup a server with -R -Fp
(the new recovery.conf was written into the target directory)
- backup a server with -R -Ftar
(the new recovery.conf was written into the base.tar)
- backup a server with -R -Ftar -z
(the new recovery.conf was written into the base.tar.gz)

Backing up a standby server without -R preserves the original recovery.conf of the
standby, it points to the standby's source server.

Backing up a standby server with -R overwrites the original recovery.conf with the new
one pointing to the standby instead of the standby's source server. Without -Ft, it is
obvious. With -Ft, there are two recovery.conf files in the tar file and upon extracting it,
the last written one (the one generated via -R) overwrites the original.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

Attachments:

01-PQconninfo-v13.patchtext/x-patch; name=01-PQconninfo-v13.patchDownload+211-71
02-pg_basebackup-v10.patchtext/x-patch; name=02-pg_basebackup-v10.patchDownload+372-2
#18Boszormenyi Zoltan
zb@cybertec.at
In reply to: Fujii Masao (#14)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

2012-10-11 02:02 keltezéssel, Fujii Masao írta:

On Thu, Oct 11, 2012 at 3:36 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

When tar output format is specified together with -R option, recovery.conf
is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

Because it's more user-friendly. If recovery.conf is not included in base.tar,
when base.tar is extracted to disk to use the backup, a user always needs
to copy recovery.conf to the extracted directory. OTOH if it's included in
base.tar, such copy operation is not required and we can simplify the
procedures to use the backup a bit.

It's implemented now.

+        setting up the standby. Since creating a backup for a standalone
+        server with <option>-x</option> or <option>-X</option> and adding
+        a recovery.conf to it wouldn't make a working standby, these
options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

I don't know. Pointers?

There is no restriction that the backup which was taken by using
pg_basebackup -x cannot be used to start the standby. So I wonder
why -R option cannot work together with -x. It's useful if recovery.conf
is automatically written even with pg_basebackup -x.

There was a problem with 9.0.x (maybe even with 9.1) that the backup
failed to come up as a standby if -x or -X was specified. I don't know
if it was a bug, limitation or intended behaviour.

Before removing the check for conflicting options, I would like to ask:
is there such a known conflict with --xlog-method=stream?

And I found another problem: when -(stdout) is specified in -D option,
recovery.conf fails to be written.

$ pg_basebackup -D- -F t -c fast -R > hoge.tar
NOTICE: WAL archiving is not enabled; you must ensure that all
required WAL segments are copied through other means to complete the
backup
pg_basebackup: cannot create -/recovery.conf: No such file or directory

Now it works with recovery.conf written into the tar itself. I also tried

$ pg_basebackup -D- -Ft -R

and the directory named "-" was created and of course the recovery.conf
inside it. Is this the intended behaviour regarding "stdout is to be treated
as a directory"?

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/

#19Fujii Masao
masao.fujii@gmail.com
In reply to: Boszormenyi Zoltan (#17)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

Thanks for updating the patch!

On Sun, Oct 14, 2012 at 8:41 PM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

Backing up a standby server without -R preserves the original recovery.conf
of the
standby, it points to the standby's source server.

Backing up a standby server with -R overwrites the original recovery.conf
with the new
one pointing to the standby instead of the standby's source server. Without
-Ft, it is
obvious. With -Ft, there are two recovery.conf files in the tar file and
upon extracting it,
the last written one (the one generated via -R) overwrites the original.

The tar file is always extracted such way in all platform which PostgreSQL
supports? I'm just concerned about that some tool in some platform might
prefer the original recovery.conf when extracting tar file. If the spec of tar
format specifies such behavior (i.e., the last written file of the same name
is always preferred), it's OK.

I found the bug that recovery.conf is included in the tar file of the tablespace
instead of base.tar, when there are tablespaces in the server.

Maybe this is nitpicky problem,,,, but...
If port number is not explicitly specified in pg_basebackup, the port
number is not
included to primary_conninfo in recovery.conf which is created during
the backup.
That is, the standby server using such recovery.conf tries to connect
to the default
port number because the port number is not supplied in primary_conninfo. This
assumes that the default port number is the same between the master and standby.
But this is not true. The default port number can be changed in --with-pgport
configure option, so the default port number might be different
between the master
and standby. To avoid this uncertainty, pg_basebackup -R should always include
the port number in primary_conninfo?

When the password is required to connect to the server, pg_basebackup -R
always writes the password setting into primary_conninfo in recovery.conf.
But if the password is supplied from .pgpass, ISTM that the password setting
doesn't need to be written into primary_conninfo. Right?

+        The password written into recovery.conf is not escaped even if special
+        characters appear in it. The administrator must review recovery.conf
+        to ensure proper escaping.

Is it difficult to make pg_basebackup escape the special characters in the
password? It's better if we can remove this restriction.

I've not reviewed PQconninfo patch yet. Will review.

Regards,

--
Fujii Masao

#20Fujii Masao
masao.fujii@gmail.com
In reply to: Boszormenyi Zoltan (#18)
Re: [PATCH] Make pg_basebackup configure and start standby [Review]

On Mon, Oct 15, 2012 at 12:57 AM, Boszormenyi Zoltan <zb@cybertec.at> wrote:

2012-10-11 02:02 keltezéssel, Fujii Masao írta:

On Thu, Oct 11, 2012 at 3:36 AM, Boszormenyi Zoltan <zb@cybertec.at>
wrote:

2012-10-10 18:23 keltezéssel, Fujii Masao írta:

When tar output format is specified together with -R option,
recovery.conf
is
not included in base.tar. I think it should.

Why? This patch only promises to write the recovery.conf into the
directory specified with -D.

Because it's more user-friendly. If recovery.conf is not included in
base.tar,
when base.tar is extracted to disk to use the backup, a user always needs
to copy recovery.conf to the extracted directory. OTOH if it's included in
base.tar, such copy operation is not required and we can simplify the
procedures to use the backup a bit.

It's implemented now.

Thanks a lot!

+        setting up the standby. Since creating a backup for a
standalone
+        server with <option>-x</option> or <option>-X</option> and
adding
+        a recovery.conf to it wouldn't make a working standby, these
options
+        naturally conflict.

Is this right? ISTM that basically we can use pg_basebackup -x to take
a base backup for starting the standby for now. No?

I don't know. Pointers?

There is no restriction that the backup which was taken by using
pg_basebackup -x cannot be used to start the standby. So I wonder
why -R option cannot work together with -x. It's useful if recovery.conf
is automatically written even with pg_basebackup -x.

There was a problem with 9.0.x (maybe even with 9.1) that the backup
failed to come up as a standby if -x or -X was specified. I don't know
if it was a bug, limitation or intended behaviour.

It sounds a bug to me. It's helpful if you provide the self-contained test case.

Before removing the check for conflicting options, I would like to ask:
is there such a known conflict with --xlog-method=stream?

AFAIK, No, we can use the backup which pg_basebackup --xlog-method=stream
took, to start the standby. BTW, --xlog-method=stream cannot be specified
together with -F tar.

And I found another problem: when -(stdout) is specified in -D option,
recovery.conf fails to be written.

$ pg_basebackup -D- -F t -c fast -R > hoge.tar
NOTICE: WAL archiving is not enabled; you must ensure that all
required WAL segments are copied through other means to complete the
backup
pg_basebackup: cannot create -/recovery.conf: No such file or directory

Now it works with recovery.conf written into the tar itself. I also tried

$ pg_basebackup -D- -Ft -R

and the directory named "-" was created and of course the recovery.conf
inside it. Is this the intended behaviour regarding "stdout is to be treated
as a directory"?

Yes. Thanks.

Regards,

--
Fujii Masao

#21Boszormenyi Zoltan
zb@cybertec.at
In reply to: Fujii Masao (#19)
#22Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#21)
#23Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#22)
#24Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#23)
#25Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#24)
#26Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Boszormenyi Zoltan (#25)
#27Magnus Hagander
magnus@hagander.net
In reply to: Alvaro Herrera (#26)
#28Magnus Hagander
magnus@hagander.net
In reply to: Magnus Hagander (#27)
#29Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#28)
#30Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#28)
#31Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#30)
#32Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#30)
#33Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#32)
#34Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Boszormenyi Zoltan (#33)
#35Boszormenyi Zoltan
zb@cybertec.at
In reply to: Alvaro Herrera (#34)
#36Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Boszormenyi Zoltan (#35)
#37Tom Lane
tgl@sss.pgh.pa.us
In reply to: Boszormenyi Zoltan (#33)
#38Boszormenyi Zoltan
zb@cybertec.at
In reply to: Tom Lane (#37)
#39Boszormenyi Zoltan
zb@cybertec.at
In reply to: Tom Lane (#37)
#40Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#39)
#41Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#40)
#42Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#40)
#43Magnus Hagander
magnus@hagander.net
In reply to: Boszormenyi Zoltan (#42)
#44Magnus Hagander
magnus@hagander.net
In reply to: Boszormenyi Zoltan (#42)
#45Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#44)
#46Magnus Hagander
magnus@hagander.net
In reply to: Boszormenyi Zoltan (#45)
#47Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#43)
#48Tom Lane
tgl@sss.pgh.pa.us
In reply to: Boszormenyi Zoltan (#47)
#49Boszormenyi Zoltan
zb@cybertec.at
In reply to: Tom Lane (#48)
#50Magnus Hagander
magnus@hagander.net
In reply to: Boszormenyi Zoltan (#49)
#51Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#50)
#52Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#51)
#53Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#50)
#54Boszormenyi Zoltan
zb@cybertec.at
In reply to: Boszormenyi Zoltan (#52)
#55Magnus Hagander
magnus@hagander.net
In reply to: Boszormenyi Zoltan (#54)
#56Magnus Hagander
magnus@hagander.net
In reply to: Magnus Hagander (#55)
#57Boszormenyi Zoltan
zb@cybertec.at
In reply to: Magnus Hagander (#56)
#58Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Boszormenyi Zoltan (#53)