Turning recovery.conf into GUCs

Started by Jaime Casanovaover 12 years ago107 messageshackers
Jump to latest
#1Jaime Casanova
jcasanov@systemguards.com.ec

Hi everyone,

Seems that the latest patch in this series is:
/messages/by-id/CAB7nPqRLWLH1b0YsvqYF-zOFjrv4FRiurQ6yqcP3wjBp=TJOLg@mail.gmail.com

= Applying =

Reading the threads it seems there is consensus in this happening, so
let's move in that direction.
The patch applies almost cleanly except for a minor change in xlog.c

= Building =

In pg_basebackup we have now 2 unused functions:
escapeConnectionParameter and escape_quotes. the only use of these
functions was when that tool created the recovery.conf file so they
aren't needed anymore.

= Functionality =

I have been testing functionality and it looks good so far, i still
need to test a few things but i don't expect anything bad.

I have 2 complaints though:

1) the file to trigger recovery is now called standby.enabled. I know
this is because we use the file to also make the node a standby.
Now, reality is that the file doesn't make the node a standby but the
combination of this file (which starts recovery) + standby_mode=on.
so, i agree with the suggestion of naming the file: recovery.trigger

2) This patch removes the dual existence of recovery.conf: as a state
file and as a configuration file

- the former is accomplished by changing the name of the file that
triggers recovery (from recovery.conf to standby.enabled)
- the latter is done by moving all parameters to postgresql.conf and
*not reading* recovery.conf anymore

so, after applying this patch postgres don't use recovery.conf for
anything... except for complaining.
it's completely fair to say we are no longer using that file and
ignoring its existence, but why we should care if users want to use a
file with that name for inclusion in postgresql.conf or where they put
that hypotetic file?

after this patch, recovery.conf will have no special meaning anymore
so let's the user put it whatever files he wants, wherever he wants.
if we really want to warn the user, use WARNING instead of FATAL

= Code & functionality =

why did you changed the context in which we can set archive_command?
please, let it as a SIGHUP parameter

-       {"archive_command", PGC_SIGHUP, WAL_ARCHIVING,
+       {"archive_command", PGC_POSTMASTER, WAL_ARCHIVING,

All parameters moved from recovery.conf has been marked as
PGC_POSTMASTER, but most of them are clearly PGC_SIGHUP candidates

+       {"restore_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"archive_cleanup_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_end_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_target_xid", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_name", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_time", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"trigger_file", PGC_POSTMASTER, REPLICATION_STANDBY,

Not sure about these ones

+       {"recovery_target_timeline", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"primary_conninfo", PGC_POSTMASTER, REPLICATION_STANDBY,

This is the only one i agree, should be PGC_POSTMASTER only

+ {"standby_mode", PGC_POSTMASTER, REPLICATION_STANDBY,

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#2Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#1)
Re: Turning recovery.conf into GUCs

Jaime,

= Building =

In pg_basebackup we have now 2 unused functions:
escapeConnectionParameter and escape_quotes. the only use of these
functions was when that tool created the recovery.conf file so they
aren't needed anymore.

Except that we'll want 9.4's -R to do something, probably create a file
called conf.d/replication.conf. Mind you, it won't need the same wonky
quoting stuff.

1) the file to trigger recovery is now called standby.enabled. I know
this is because we use the file to also make the node a standby.
Now, reality is that the file doesn't make the node a standby but the
combination of this file (which starts recovery) + standby_mode=on.
so, i agree with the suggestion of naming the file: recovery.trigger

2) This patch removes the dual existence of recovery.conf: as a state
file and as a configuration file

- the former is accomplished by changing the name of the file that
triggers recovery (from recovery.conf to standby.enabled)
- the latter is done by moving all parameters to postgresql.conf and
*not reading* recovery.conf anymore

so, after applying this patch postgres don't use recovery.conf for
anything... except for complaining.
it's completely fair to say we are no longer using that file and
ignoring its existence, but why we should care if users want to use a
file with that name for inclusion in postgresql.conf or where they put
that hypotetic file?

So this is a bit of a catch-22. If we allow the user to use a file
named "recovery.conf" in $PGDATA, then the user may be unaware that the
*meaning* of the file has changed, which can result in unplanned
promotion of replicas after upgrade.

*on the other hand*, if we prevent creation of a configuration file
named "recovery.conf", then we block efforts to write
backwards-compatible management utilities.

AFAIK, there is no good solution for this, which is why it's taken so
long for us to resolve the general issue. Given that, I think it's
better to go for the breakage and all the warnings. If a user wants a
file called recovery.conf, put it in the conf.d directory.

I don't remember, though: how does this patch handle PITR recovery?

= Code & functionality =

+       {"restore_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"archive_cleanup_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_end_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_target_xid", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_name", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_time", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"trigger_file", PGC_POSTMASTER, REPLICATION_STANDBY,

Not sure about these ones

+       {"recovery_target_timeline", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"primary_conninfo", PGC_POSTMASTER, REPLICATION_STANDBY,

It would be really nice to change these on the fly; it would help a lot
of issues with minor changes to replication config. I can understand,
though, that the replication code might not be prepared for that.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#3Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#2)
Re: Turning recovery.conf into GUCs

On Wed, Oct 16, 2013 at 1:34 PM, Josh Berkus <josh@agliodbs.com> wrote:

Jaime,

= Building =

In pg_basebackup we have now 2 unused functions:
escapeConnectionParameter and escape_quotes. the only use of these
functions was when that tool created the recovery.conf file so they
aren't needed anymore.

Except that we'll want 9.4's -R to do something, probably create a file
called conf.d/replication.conf. Mind you, it won't need the same wonky
quoting stuff.

Currently the patch uses -R to create the recovery trigger file

1) the file to trigger recovery is now called standby.enabled. I know
this is because we use the file to also make the node a standby.
Now, reality is that the file doesn't make the node a standby but the
combination of this file (which starts recovery) + standby_mode=on.
so, i agree with the suggestion of naming the file: recovery.trigger

2) This patch removes the dual existence of recovery.conf: as a state
file and as a configuration file

- the former is accomplished by changing the name of the file that
triggers recovery (from recovery.conf to standby.enabled)
- the latter is done by moving all parameters to postgresql.conf and
*not reading* recovery.conf anymore

so, after applying this patch postgres don't use recovery.conf for
anything... except for complaining.
it's completely fair to say we are no longer using that file and
ignoring its existence, but why we should care if users want to use a
file with that name for inclusion in postgresql.conf or where they put
that hypotetic file?

So this is a bit of a catch-22. If we allow the user to use a file
named "recovery.conf" in $PGDATA, then the user may be unaware that the
*meaning* of the file has changed, which can result in unplanned
promotion of replicas after upgrade.

well, after upgrade you should do checks. and even if it happens,
after it happens once people will be aware of the change.
now, some suggestions were made to avoid the problem. 1) read the file
if exists last in the process of postgresql.conf, 2) add a GUC
indicating if it should read it and include it (not using it as a
trigger file). another one, 3) include in this release an
include_if_exists directive and give a warning if it sees the file
then include it, on next release remove the include_if_exists (at
least that way people will be warned before breaking compatibility)

please, keep in mind none of these suggestions include make the
recovery.conf file act as a trigger for recovery

*on the other hand*, if we prevent creation of a configuration file
named "recovery.conf", then we block efforts to write
backwards-compatible management utilities.

and all tools and procedures that currently exists.

AFAIK, there is no good solution for this, which is why it's taken so
long for us to resolve the general issue. Given that, I think it's
better to go for the breakage and all the warnings. If a user wants a
file called recovery.conf, put it in the conf.d directory.

well, there should be good solutions... maybe we haven't thought them yet.
anyway, we can't postpone the decision forever. we need to make a
decision and stick with it otherwise this patch will be stalled N
releases for no good reason

I don't remember, though: how does this patch handle PITR recovery?

exactly as it is now, if it sees the recovery trigger file, then it
starts ArchiveRecovery and after it finish delete the file (the only
difference) and increment the timeline

= Code & functionality =

+       {"restore_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"archive_cleanup_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_end_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_target_xid", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_name", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_time", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"trigger_file", PGC_POSTMASTER, REPLICATION_STANDBY,

Not sure about these ones

+       {"recovery_target_timeline", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"primary_conninfo", PGC_POSTMASTER, REPLICATION_STANDBY,

It would be really nice to change these on the fly; it would help a lot
of issues with minor changes to replication config. I can understand,
though, that the replication code might not be prepared for that.

well, archive_command can be changed right now with a SIGHUP so at
least that one shouldn't change... and i don't think most of these are
too different. even if we are not sure we can do this now and change
them as SIGHUP later

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#4Michael Paquier
michael@paquier.xyz
In reply to: Jaime Casanova (#3)
Re: Turning recovery.conf into GUCs

On Fri, Oct 18, 2013 at 1:13 PM, Jaime Casanova <jaime@2ndquadrant.com> wrote:

= Code & functionality =

+       {"restore_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"archive_cleanup_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_end_command", PGC_POSTMASTER, WAL_ARCHIVE_RECOVERY,
+       {"recovery_target_xid", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_name", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"recovery_target_time", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"trigger_file", PGC_POSTMASTER, REPLICATION_STANDBY,

Not sure about these ones

+       {"recovery_target_timeline", PGC_POSTMASTER, WAL_RECOVERY_TARGET,
+       {"primary_conninfo", PGC_POSTMASTER, REPLICATION_STANDBY,

It would be really nice to change these on the fly; it would help a lot
of issues with minor changes to replication config. I can understand,
though, that the replication code might not be prepared for that.

well, archive_command can be changed right now with a SIGHUP so at
least that one shouldn't change... and i don't think most of these are
too different. even if we are not sure we can do this now and change
them as SIGHUP later

Changing those parameters don't really matter as long as the node is
not performing a recovery IMO, but I'd rather see a careful approach
here and let all those parameters as PGC_POSTMASTER for now to avoid
any surprises. Perhaps a second patch on top of this one could be the
addition of context name like SIGHUP_RECOVERY, aka just allow those
parameters to be updated with SIGHUP as long as the node is not in
recovery.
--
Michael

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#5Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#1)
Re: Turning recovery.conf into GUCs

Jaime,

Except that we'll want 9.4's -R to do something, probably create a file
called conf.d/replication.conf. Mind you, it won't need the same wonky
quoting stuff.

Currently the patch uses -R to create the recovery trigger file

Right, I'm saying that we'll want to do better than that for release,
but that's dependant on committing the conf directory patch.

Note that this change makes committing the conf.d patch extra-important;
it's going to be a LOT easier to upgrade tools for 9.4 if we have that.

well, after upgrade you should do checks. and even if it happens,
after it happens once people will be aware of the change.
now, some suggestions were made to avoid the problem. 1) read the file
if exists last in the process of postgresql.conf, 2) add a GUC
indicating if it should read it and include it (not using it as a
trigger file). another one, 3) include in this release an
include_if_exists directive and give a warning if it sees the file
then include it, on next release remove the include_if_exists (at
least that way people will be warned before breaking compatibility)

I think all of these suggestions just make our code more complicated
without improving the upgrade situation.

The reason given (and I think it's pretty good) for erroring on
recovery.conf is that we don't want people to accidentally take a server
out of replication because they didn't check which version of PostgreSQL
they are on.

*on the other hand*, if we prevent creation of a configuration file
named "recovery.conf", then we block efforts to write
backwards-compatible management utilities.

and all tools and procedures that currently exists.

Right. However, exploring your suggestions above, none of those
workarounds prevent breakage. And in some cases, they make the breakage
less obvious than the current patch does. If repmgr 1.2 / OmniPITR 1.2
won't work correctly with 9.4, then we want those tools to break at
upgrade time, not later when the user is trying to fail over.

For that matter, 9.4 is a very good time (relatively speaking) to break
replication tools because the new logical replication is going to cause
everyone to rev their tools anyway.

This kind of breakage alone might end up being a good reason to call the
next version 10.0.

well, there should be good solutions... maybe we haven't thought them yet.
anyway, we can't postpone the decision forever. we need to make a
decision and stick with it otherwise this patch will be stalled N
releases for no good reason

I think if there were a good solution, sometime in the last 1.5 years
someone would have suggested it. Gods know Simon has tried.

exactly as it is now, if it sees the recovery trigger file, then it
starts ArchiveRecovery and after it finish delete the file (the only
difference) and increment the timeline

OK, so if I'm doing a PITR recovery, I just put the recovery variables
into postgresql.conf, then?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#6Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#5)
Re: Turning recovery.conf into GUCs

On Fri, Oct 18, 2013 at 11:32 AM, Josh Berkus <josh@agliodbs.com> wrote:

exactly as it is now, if it sees the recovery trigger file, then it
starts ArchiveRecovery and after it finish delete the file (the only
difference) and increment the timeline

OK, so if I'm doing a PITR recovery, I just put the recovery variables
into postgresql.conf, then?

create a recovery trigger file (called standby.enabled in current
patch) in $PGDATA and start the server

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#7Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#5)
Re: Turning recovery.conf into GUCs

On 2013-10-18 09:32:15 -0700, Josh Berkus wrote:

For that matter, 9.4 is a very good time (relatively speaking) to break
replication tools because the new logical replication is going to cause
everyone to rev their tools anyway.

We're hopefully getting changeset extraction in, but there's little
chance of a full blown replication solution making it in in 9.4...

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#8Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#1)
Re: Turning recovery.conf into GUCs

On 10/18/2013 12:29 PM, Andres Freund wrote:

On 2013-10-18 09:32:15 -0700, Josh Berkus wrote:

For that matter, 9.4 is a very good time (relatively speaking) to break
replication tools because the new logical replication is going to cause
everyone to rev their tools anyway.

We're hopefully getting changeset extraction in, but there's little
chance of a full blown replication solution making it in in 9.4...

I thought changeset extraction was the only thing going into core? What
else do we need?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#9Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#8)
Re: Turning recovery.conf into GUCs

On 2013-10-18 13:16:52 -0700, Josh Berkus wrote:

On 10/18/2013 12:29 PM, Andres Freund wrote:

On 2013-10-18 09:32:15 -0700, Josh Berkus wrote:

For that matter, 9.4 is a very good time (relatively speaking) to break
replication tools because the new logical replication is going to cause
everyone to rev their tools anyway.

We're hopefully getting changeset extraction in, but there's little
chance of a full blown replication solution making it in in 9.4...

I thought changeset extraction was the only thing going into core? What
else do we need?

Well, I personally want more in core mid/long term, but anyway.

Without released, proven and stable logical in-core replication
technology using this, I don't see why repmgr or something related would
need/want to change?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#10Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#1)
Re: Turning recovery.conf into GUCs

On 10/18/2013 01:35 PM, Andres Freund wrote:

On 2013-10-18 13:16:52 -0700, Josh Berkus wrote:

I thought changeset extraction was the only thing going into core? What
else do we need?

Well, I personally want more in core mid/long term, but anyway.

I've lost track of the plan, then.

Hmmm ... we need replication of DDL commands, no?

Without released, proven and stable logical in-core replication
technology using this, I don't see why repmgr or something related would
need/want to change?

Repmgr is designed to manage binary replication, not perform it.

What will likely change first is Slony and Bucardo, who have a strong
interest in dumping triggers and queues. A contrib module which did the
simplest implementation -- that is, whole-database M-S replication --
would also be a good idea, especially since it would provide an example
of how to build your own.

But I'd be wary of going beyond that in core, because you very quickly
get into the territory of trying to satisfy multiple exclusive
use-cases. Let's focus on providing a really good API which enables
people to build their own tools.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#11Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#10)
Re: Turning recovery.conf into GUCs

On 2013-10-18 14:16:04 -0700, Josh Berkus wrote:

On 10/18/2013 01:35 PM, Andres Freund wrote:

On 2013-10-18 13:16:52 -0700, Josh Berkus wrote:

I thought changeset extraction was the only thing going into core? What
else do we need?

Well, I personally want more in core mid/long term, but anyway.

I've lost track of the plan, then.

Hmmm ... we need replication of DDL commands, no?

Without released, proven and stable logical in-core replication
technology using this, I don't see why repmgr or something related would
need/want to change?

Repmgr is designed to manage binary replication, not perform it.

Obviously.

What will likely change first is Slony and Bucardo, who have a strong
interest in dumping triggers and queues.

But I don't understand what that has to do with recovery.conf and
breakage around it.

A contrib module which did the
simplest implementation -- that is, whole-database M-S replication --
would also be a good idea, especially since it would provide an example
of how to build your own.

But I'd be wary of going beyond that in core, because you very quickly
get into the territory of trying to satisfy multiple exclusive
use-cases. Let's focus on providing a really good API which enables
people to build their own tools.

We'll see. I am certain we'll have many discussions about the bits and
pieces you need to build a great replication solution (of which we imo
don't have any yet).

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#12Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#5)
Re: Turning recovery.conf into GUCs

On Fri, Oct 18, 2013 at 11:32 AM, Josh Berkus <josh@agliodbs.com> wrote:

Jaime,

well, after upgrade you should do checks. and even if it happens,
after it happens once people will be aware of the change.
now, some suggestions were made to avoid the problem. 1) read the file
if exists last in the process of postgresql.conf, 2) add a GUC
indicating if it should read it and include it (not using it as a
trigger file). another one, 3) include in this release an
include_if_exists directive and give a warning if it sees the file
then include it, on next release remove the include_if_exists (at
least that way people will be warned before breaking compatibility)

I think all of these suggestions just make our code more complicated
without improving the upgrade situation.

well #3 just add a line in postgresql.conf (an include_if_exists) and
current patch gives an error in case it finds the file (i'm suggesting
to make it a warning instead).
how does that makes our code more complicated?

The reason given (and I think it's pretty good) for erroring on
recovery.conf is that we don't want people to accidentally take a server
out of replication because they didn't check which version of PostgreSQL
they are on.

well, people will go out of replication also if they forgot the
recovery trigger file
even if they set the variables in postgresql.conf

it happens two me twice today ;)

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#13Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#1)
Re: Turning recovery.conf into GUCs

On 10/18/2013 02:58 PM, Jaime Casanova wrote:

well #3 just add a line in postgresql.conf (an include_if_exists) and
current patch gives an error in case it finds the file (i'm suggesting
to make it a warning instead).
how does that makes our code more complicated?

Well, that's a couple extra lines only, I know. However, it doesn't
actually help with the breakage any, since recovery.conf *still* won't
work as a trigger file.

The only thing which would prevent breakage (proposed by Simon, I think)
is having recovery.conf have an include_if_exists, AND have
recovery.conf be an 'alternate' name for replication.trigger. However,
even this would break, and in IMHO ways which would tend to happen at
failover time rather than upgrade time.

To put it clearly: if we're going to have breakage, I want it to be at
upgrade time, when the database is *already down*, and not at failover
time or some other time when downtime is not planned.

well, people will go out of replication also if they forgot the
recovery trigger file
even if they set the variables in postgresql.conf

it happens two me twice today ;)

Right. What I'd like to avoid is having folks try to use, for example,
repmgr 1.2 with PostgreSQL 9.4 and have their replication break and them
not notice for a couple hours of operation. I'd rather have PostgreSQL
9.4 refuse to come up, so that they know *immediately* that something is
wrong.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#14Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#2)
Re: Turning recovery.conf into GUCs

On 10/18/2013 02:36 PM, Andres Freund wrote:

What will likely change first is Slony and Bucardo, who have a strong
interest in dumping triggers and queues.

But I don't understand what that has to do with recovery.conf and
breakage around it.

The simple thinking is this: if we announce and promote new replication,
then our users who do upgrade are going to expect to upgrade their
replication tools at the same time, even if they're not using the new
replication. That is people will look for a repmgr 2.0 / OmniPITR 1.5
and update to it.

Now, as a tool author, I know that supporting both models is going to be
annoying. But necessary.

And, as I said before, we need to do the GUC merger in the same release
we introduce configuration directory (or after it).

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#15Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#14)
Re: Turning recovery.conf into GUCs

On Mon, Oct 21, 2013 at 11:53 AM, Josh Berkus <josh@agliodbs.com> wrote:

On 10/18/2013 02:36 PM, Andres Freund wrote:

What will likely change first is Slony and Bucardo, who have a strong
interest in dumping triggers and queues.

But I don't understand what that has to do with recovery.conf and
breakage around it.

The simple thinking is this: if we announce and promote new replication,
then our users who do upgrade are going to expect to upgrade their
replication tools at the same time, even if they're not using the new
replication. That is people will look for a repmgr 2.0 / OmniPITR 1.5
and update to it.

Now, as a tool author, I know that supporting both models is going to be
annoying. But necessary.

AFAIU, even if we get in all about logical replication today that
won't affect tools that manage binary replication.

And, as I said before, we need to do the GUC merger in the same release
we introduce configuration directory (or after it).

you mean the ALTER SYSTEM syntax? anyway, why that restriction?

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#16Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#2)
Re: Turning recovery.conf into GUCs

Jaime,

And, as I said before, we need to do the GUC merger in the same release
we introduce configuration directory (or after it).

you mean the ALTER SYSTEM syntax? anyway, why that restriction?

No, I'm referring to the proposal to have an automatically created and
included conf.d directory for extra configuration files.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#17Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#16)
Re: Turning recovery.conf into GUCs

On Mon, Oct 21, 2013 at 2:57 PM, Josh Berkus <josh@agliodbs.com> wrote:

Jaime,

And, as I said before, we need to do the GUC merger in the same release
we introduce configuration directory (or after it).

you mean the ALTER SYSTEM syntax? anyway, why that restriction?

No, I'm referring to the proposal to have an automatically created and
included conf.d directory for extra configuration files.

9.3 has include_dir and postgresql.conf has this setted:
"""
#include_dir = 'conf.d' # include files ending in '.conf' from
# directory 'conf.d'
"""

anything before this should be up to the packagers, no?

or, do you mean something else?

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#18Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#2)
Re: Turning recovery.conf into GUCs

9.3 has include_dir and postgresql.conf has this setted:
"""
#include_dir = 'conf.d' # include files ending in '.conf' from
# directory 'conf.d'
"""

anything before this should be up to the packagers, no?

or, do you mean something else?

Well, I was just pointing out that it would make things *much* easier
for tool authors if conf.d were enabled by default, instead of disabled
by default. Then they could change tools to write a "recovery.conf"
file to conf.d.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#19Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#18)
Re: Turning recovery.conf into GUCs

On Mon, Oct 21, 2013 at 3:57 PM, Josh Berkus <josh@agliodbs.com> wrote:

9.3 has include_dir and postgresql.conf has this setted:
"""
#include_dir = 'conf.d' # include files ending in '.conf' from
# directory 'conf.d'
"""

anything before this should be up to the packagers, no?

or, do you mean something else?

Well, I was just pointing out that it would make things *much* easier
for tool authors if conf.d were enabled by default, instead of disabled
by default. Then they could change tools to write a "recovery.conf"
file to conf.d.

I agree, it would be much easier, but that is not relevant to this patch.

I have rebased Michael Paquier's patch and did a few changes:

* changed standby.enabled filename to recovery.trigger
* make archive_command a SIGHUP parameter again
* make restore_command a SIGHUP parameter
* rename restore_command variable to XLogRestoreCommand to match
XLogArchiveCommand

we can also make primary_conninfo a SIGHUP parameter, of course the
DBA will need to force a reconection after that.
after this is done we can look at the other recovery parameters.

--
Jaime Casanova www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación
Phone: +593 4 5107566 Cell: +593 987171157

Attachments:

recovery_guc_v5.patchtext/x-patch; charset=US-ASCII; name=recovery_guc_v5.patchDownload+905-960
#20Peter Eisentraut
peter_e@gmx.net
In reply to: Jaime Casanova (#19)
Re: Turning recovery.conf into GUCs

On 11/13/13, 12:17 AM, Jaime Casanova wrote:

I have rebased Michael Paquier's patch and did a few changes:

* changed standby.enabled filename to recovery.trigger
* make archive_command a SIGHUP parameter again
* make restore_command a SIGHUP parameter
* rename restore_command variable to XLogRestoreCommand to match
XLogArchiveCommand

Please check for compiler warnings in pg_basebackup:

pg_basebackup.c:1109:1: warning: �escapeConnectionParameter� defined but not used [-Wunused-function]
pg_basebackup.c:1168:1: warning: �escape_quotes� defined but not used [-Wunused-function]

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#21Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Peter Eisentraut (#20)
#22Michael Paquier
michael@paquier.xyz
In reply to: Jaime Casanova (#21)
#23Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#2)
#24Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#23)
#25Andres Freund
andres@anarazel.de
In reply to: Jaime Casanova (#24)
#26Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#25)
#27Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#26)
#28Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andres Freund (#27)
#29Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Andres Freund (#27)
#30Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#28)
#31Andres Freund
andres@anarazel.de
In reply to: Jaime Casanova (#29)
#32Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Andres Freund (#25)
#33Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Jaime Casanova (#32)
#34Robert Haas
robertmhaas@gmail.com
In reply to: Jaime Casanova (#33)
#35Andres Freund
andres@anarazel.de
In reply to: Jaime Casanova (#32)
#36Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#35)
#37Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#18)
#38Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#37)
#39Joshua D. Drake
jd@commandprompt.com
In reply to: Andres Freund (#38)
#40Alex Shulgin
ash@commandprompt.com
In reply to: Andres Freund (#35)
#41Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#18)
#42Stephen Frost
sfrost@snowman.net
In reply to: Josh Berkus (#41)
#43Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#21)
#44Amit Kapila
amit.kapila16@gmail.com
In reply to: Stephen Frost (#42)
#45Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#43)
#46Alex Shulgin
ash@commandprompt.com
In reply to: Alex Shulgin (#40)
#47Alex Shulgin
ash@commandprompt.com
In reply to: Josh Berkus (#41)
#48Stephen Frost
sfrost@snowman.net
In reply to: Amit Kapila (#44)
#49Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Stephen Frost (#48)
#50Stephen Frost
sfrost@snowman.net
In reply to: Alvaro Herrera (#49)
#51Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Stephen Frost (#50)
#52Stephen Frost
sfrost@snowman.net
In reply to: Alvaro Herrera (#51)
#53Andres Freund
andres@anarazel.de
In reply to: Stephen Frost (#50)
#54Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alex Shulgin (#46)
#55David G. Johnston
david.g.johnston@gmail.com
In reply to: Stephen Frost (#50)
#56Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Josh Berkus (#41)
#57Jaime Casanova
jcasanov@systemguards.com.ec
In reply to: Jaime Casanova (#56)
#58Josh Berkus
josh@agliodbs.com
In reply to: Peter Eisentraut (#20)
#59Josh Berkus
josh@agliodbs.com
In reply to: Andres Freund (#35)
#60Stephen Frost
sfrost@snowman.net
In reply to: Josh Berkus (#59)
#61Alex Shulgin
ash@commandprompt.com
In reply to: Jaime Casanova (#56)
#62Alex Shulgin
ash@commandprompt.com
In reply to: Josh Berkus (#58)
#63Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#58)
#64Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#23)
#65Alex Shulgin
ash@commandprompt.com
In reply to: Josh Berkus (#64)
#66Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#32)
#67Alex Shulgin
ash@commandprompt.com
In reply to: Alex Shulgin (#40)
#68Michael Paquier
michael@paquier.xyz
In reply to: Alex Shulgin (#67)
#69Alex Shulgin
ash@commandprompt.com
In reply to: Michael Paquier (#68)
#70Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#18)
#71Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Josh Berkus (#70)
#72Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#24)
#73Andres Freund
andres@anarazel.de
In reply to: Alex Shulgin (#69)
#74Alex Shulgin
ash@commandprompt.com
In reply to: Andres Freund (#73)
#75Alex Shulgin
ash@commandprompt.com
In reply to: Alex Shulgin (#67)
#76Petr Jelinek
petr@2ndquadrant.com
In reply to: Alex Shulgin (#75)
#77Alex Shulgin
ash@commandprompt.com
In reply to: Petr Jelinek (#76)
#78Alex Shulgin
ash@commandprompt.com
In reply to: Alex Shulgin (#77)
#79Petr Jelinek
petr@2ndquadrant.com
In reply to: Alex Shulgin (#78)
#80Peter Eisentraut
peter_e@gmx.net
In reply to: Alex Shulgin (#75)
#81Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#18)
#82Mark Kirkwood
mark.kirkwood@catalyst.net.nz
In reply to: Josh Berkus (#81)
#83Peter Eisentraut
peter_e@gmx.net
In reply to: Josh Berkus (#81)
#84Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#21)
#85Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#23)
#86Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#85)
#87Josh Berkus
josh@agliodbs.com
In reply to: Andres Freund (#35)
#88Andres Freund
andres@anarazel.de
In reply to: Peter Eisentraut (#80)
#89Andres Freund
andres@anarazel.de
In reply to: Josh Berkus (#87)
#90Josh Berkus
josh@agliodbs.com
In reply to: Alex Shulgin (#75)
#91Kevin Grittner
Kevin.Grittner@wicourts.gov
In reply to: Andres Freund (#89)
#92Peter Eisentraut
peter_e@gmx.net
In reply to: Josh Berkus (#85)
#93Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#23)
#94Peter Eisentraut
peter_e@gmx.net
In reply to: Andres Freund (#88)
#95Peter Eisentraut
peter_e@gmx.net
In reply to: Josh Berkus (#84)
#96Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#23)
#97Robert Haas
robertmhaas@gmail.com
In reply to: Peter Eisentraut (#80)
#98Petr Jelinek
petr@2ndquadrant.com
In reply to: Robert Haas (#97)
#99Josh Berkus
josh@agliodbs.com
In reply to: Jaime Casanova (#19)
#100Michael Paquier
michael@paquier.xyz
In reply to: Peter Eisentraut (#83)
#101Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#100)
#102Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#101)
#103Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#102)
#104Peter Eisentraut
peter_e@gmx.net
In reply to: Peter Eisentraut (#83)
#105Josh Berkus
josh@agliodbs.com
In reply to: Josh Berkus (#23)
#106Peter Eisentraut
peter_e@gmx.net
In reply to: Josh Berkus (#105)
#107Michael Paquier
michael@paquier.xyz
In reply to: Peter Eisentraut (#106)