New trigger option of pg_standby
Hi,
Current pg_standby is dangerous because the presence of the trigger
file causes recovery to end whether or not the next WAL file is available.
So, some *available* transactions may be lost at failover. Such danger
will become high if the standby server has not caught up with the primary.
Attached patch fixes the above problem by adding a new trigger option
to pg_standby; the presence of this new trigger file causes recovery to
end after replaying all the available WAL files. Specifically, pg_standby
acts like 'cp' or 'ln' command while this new trigger file exists.
I've not changed any existing features, so backward-compatibility is
maintained.
Thought?
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
Attachments:
pgstandby_new_trigger_0325.patchapplication/octet-stream; name=pgstandby_new_trigger_0325.patchDownload+195-172
On Wed, Mar 25, 2009 at 7:29 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
Attached patch fixes the above problem by adding a new trigger option
to pg_standby; the presence of this new trigger file causes recovery to
end after replaying all the available WAL files.
Shouldn't it be the default? It seems like the most expected behaviour to me.
--
Guillaume
Hi,
Thanks for the comment.
On Wed, Mar 25, 2009 at 3:50 PM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:
On Wed, Mar 25, 2009 at 7:29 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
Attached patch fixes the above problem by adding a new trigger option
to pg_standby; the presence of this new trigger file causes recovery to
end after replaying all the available WAL files.Shouldn't it be the default? It seems like the most expected behaviour to me.
Yeah, I agree... but there may be scripts for warm-standby based on
the existing default behavior. So, I didn't make a new trigger the default.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Wed, Mar 25, 2009 at 9:44 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
Yeah, I agree... but there may be scripts for warm-standby based on
the existing default behavior. So, I didn't make a new trigger the default.
I don't use pg_standby personnaly but I admit I'm quite surprised by
the current behaviour. I'm pretty sure a lot of the current users
would be surprised too.
--
Guillaume
Hi,
On Wed, Mar 25, 2009 at 5:55 PM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:
On Wed, Mar 25, 2009 at 9:44 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
Yeah, I agree... but there may be scripts for warm-standby based on
the existing default behavior. So, I didn't make a new trigger the default.I don't use pg_standby personnaly but I admit I'm quite surprised by
the current behaviour. I'm pretty sure a lot of the current users
would be surprised too.
The current behavior is documented as follows, so it may be
taken for granted by some users. I think that we shouldn't
ignore such users.
---------------
Specify a trigger file whose presence should cause recovery to
end whether or not the next WAL file is available.
---------------
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
Guillaume Smet <guillaume.smet@gmail.com> wrote:
On Wed, Mar 25, 2009 at 9:44 AM, Fujii Masao <masao.fujii@gmail.com>
wrote:
Yeah, I agree... but there may be scripts for warm-standby based on
the existing default behavior. So, I didn't make a new trigger the
default.
I don't use pg_standby personnaly but I admit I'm quite surprised by
the current behaviour. I'm pretty sure a lot of the current users
would be surprised too.
I find it hard to imagine a use case for the existing default
behavior.
-Kevin
On Wed, Mar 25, 2009 at 2:59 PM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:
I find it hard to imagine a use case for the existing default
behavior.
I thought a bit about it and I think it can be useful when your
priority is the availability of the service and you don't consider a
data loss that important: even if you have a lot of WALs segments to
replay, you may want to have your service up immediately in case of a
major problem.
Keeping it is a good idea IMHO but I don't think it should be the default.
--
Guillaume
Hi,
On Thu, Mar 26, 2009 at 12:48 AM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:
On Wed, Mar 25, 2009 at 2:59 PM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:I find it hard to imagine a use case for the existing default
behavior.I thought a bit about it and I think it can be useful when your
priority is the availability of the service and you don't consider a
data loss that important: even if you have a lot of WALs segments to
replay, you may want to have your service up immediately in case of a
major problem.
Yes, I also think that this is likely use case.
Keeping it is a good idea IMHO but I don't think it should be the default.
What does "the default" mean? You mean that new trigger should use
the existing trigger option character (-t)?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
Fujii Masao wrote:
On Thu, Mar 26, 2009 at 12:48 AM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:On Wed, Mar 25, 2009 at 2:59 PM, Kevin Grittner
<Kevin.Grittner@wicourts.gov> wrote:I find it hard to imagine a use case for the existing default
behavior.I thought a bit about it and I think it can be useful when your
priority is the availability of the service and you don't consider a
data loss that important: even if you have a lot of WALs segments to
replay, you may want to have your service up immediately in case of a
major problem.Yes, I also think that this is likely use case.
Keeping it is a good idea IMHO but I don't think it should be the default.
What does "the default" mean? You mean that new trigger should use
the existing trigger option character (-t)?
The existing behavior doesn't seem very useful to me either. Assuming
there is a use case though, we probably need to support both at the same
time, perhaps using different trigger files. If there's a use case for
both, conceivably someone will want to sometimes trigger the failover
immediately and sometimes after all WAL segments have been replayed.
Whatever we do, the signaling method to trigger failover should behave
the same.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Thu, Mar 26, 2009 at 2:51 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
What does "the default" mean? You mean that new trigger should use
the existing trigger option character (-t)?
Yes, that's my point.
I understand it seems weird to switch the options but I'm pretty sure
a lot of persons currently using -t would be surprised by the current
behaviour. Moreover playing all the remaining WALs before starting up
should be the most natural option when people are looking in the help.
That said, it would be nice to hear from people really using
pg_standby to know if they understand how it works now and if it's
what they intended when they set it up.
--
Guillaume
Hi,
Guillaume Smet wrote:
On Thu, Mar 26, 2009 at 2:51 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
What does "the default" mean? You mean that new trigger should use
the existing trigger option character (-t)?Yes, that's my point.
I understand it seems weird to switch the options but I'm pretty sure
a lot of persons currently using -t would be surprised by the current
behaviour. Moreover playing all the remaining WALs before starting up
should be the most natural option when people are looking in the help.That said, it would be nice to hear from people really using
pg_standby to know if they understand how it works now and if it's
what they intended when they set it up.
My fault not RTFM well enough, but I was surprised finding out that -t
is working like that.
+1 for me to switch -t to the new behaviour.
Cheers
--
Matteo Beccati
OpenX - http://www.openx.org
On Thu, 2009-03-26 at 08:32 +0100, Guillaume Smet wrote:
On Thu, Mar 26, 2009 at 2:51 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
What does "the default" mean? You mean that new trigger should use
the existing trigger option character (-t)?Yes, that's my point.
I understand it seems weird to switch the options but I'm pretty sure
a lot of persons currently using -t would be surprised by the current
behaviour. Moreover playing all the remaining WALs before starting up
should be the most natural option when people are looking in the help.
If the standby has fallen behind then waiting for it to catch up might
take hours to failover if it waits for all files. If you haven't been
monitoring it correctly, you have no clue. That is also a surprising
thing, so let's not jump from one surprising thing into the arms of
another.
If we go with this, I would suggest we make *neither* the default by
removing -t, and adopting two new options: something like -f == fast
failover, -p == patient failover. This then forces people to read and
understand the difference between the two behaviours so they can make an
informed choice of how they would like to act at this critical point in
time. It is justifiable because there is no single thing called a
trigger file any longer and the concept will lead to pain.
Earlier, we discussed having a single trigger file that contains an
option rather than two distinct trigger files. That design is better
because it allows the user to choose at failover time, rather than
making a binding decision at config time. That solution would be the
ideal one, IMHO, because it gives user more choice - and would allow us
to keep the -t option meaningfully. In that case the default should be
patience.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Hi Simon.
On Thu, Mar 26, 2009 at 11:50 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
Earlier, we discussed having a single trigger file that contains an
option rather than two distinct trigger files. That design is better
because it allows the user to choose at failover time, rather than
making a binding decision at config time. That solution would be the
ideal one, IMHO, because it gives user more choice - and would allow us
to keep the -t option meaningfully. In that case the default should be
patience.
Or you can define both files in your command line to have the choice.
I like the idea of removing -t and adding 2 new options so that people
are warned about the intended behavior.
Anyway, I don't have a strong opinion about how we should fix it as I
don't use pg_standby personnally, just that we should. The two options
you mention have their own merits.
--
Guillaume
Hi,
On Thu, Mar 26, 2009 at 8:54 PM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:
Hi Simon.
On Thu, Mar 26, 2009 at 11:50 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
Earlier, we discussed having a single trigger file that contains an
option rather than two distinct trigger files. That design is better
because it allows the user to choose at failover time, rather than
making a binding decision at config time. That solution would be the
ideal one, IMHO, because it gives user more choice - and would allow us
to keep the -t option meaningfully. In that case the default should be
patience.Or you can define both files in your command line to have the choice.
Personally I like this.
I like the idea of removing -t and adding 2 new options so that people
are warned about the intended behavior.
OK, I'll change the patch as Simon suggested; removing -t and adding
two new options: -f = fast failover (existing behavior), -p patient failover.
Also I'll default the patient failover, so it's performed when the signal
(SIGINT or SIGUSR1) is received.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Fri, Mar 27, 2009 at 3:38 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
OK, I'll change the patch as Simon suggested; removing -t and adding
two new options: -f = fast failover (existing behavior), -p patient failover.
Also I'll default the patient failover, so it's performed when the signal
(SIGINT or SIGUSR1) is received.
I'm wondering if we should consider backpatching this one. Even if the
feature works as advertised in the documentation.
It's a very surprising behaviour and I'm pretty sure someone will
shoot himself in the foot with it, if not already done.
Considering backpatching might change the way we want to fix it.
--
Guillaume
Simon Riggs wrote:
If we go with this, I would suggest we make *neither* the default by
removing -t, and adopting two new options: something like -f == fast
failover, -p == patient failover.
-m smart|fast|immediate :-)
On Fri, 2009-03-27 at 13:56 +0200, Peter Eisentraut wrote:
Simon Riggs wrote:
If we go with this, I would suggest we make *neither* the default by
removing -t, and adopting two new options: something like -f == fast
failover, -p == patient failover.-m smart|fast|immediate :-)
Yes, a better suggestion.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
On Fri, Mar 27, 2009 at 12:56 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
Simon Riggs wrote:
If we go with this, I would suggest we make *neither* the default by
removing -t, and adopting two new options: something like -f == fast
failover, -p == patient failover.-m smart|fast|immediate :-)
The advantage of having 2 options (or the ability to put a string
value in the trigger file) is that you can choose the behaviour when
you need to trigger it (you just have to use the 2 options with 2
different filenames). I don't think it's the case with your proposal.
--
Guillaume
Fujii Masao wrote:
On Thu, Mar 26, 2009 at 8:54 PM, Guillaume Smet
<guillaume.smet@gmail.com> wrote:On Thu, Mar 26, 2009 at 11:50 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
I like the idea of removing -t and adding 2 new options so that people
are warned about the intended behavior.OK, I'll change the patch as Simon suggested; removing -t and adding
two new options: -f = fast failover (existing behavior), -p patient failover.
Also I'll default the patient failover, so it's performed when the signal
(SIGINT or SIGUSR1) is received.
Uh oh, that's going to be quite tricky with signals. Remember that
pg_standby is called for each file. A trigger file persists until it's
deleted, but a signal will only be received by the pg_standby instance
that happens to be running at the time.
Makes me wonder if the trigger pg_standby with signals is reliable to
begin with. What if the backend is just processing a file when the
signal is fired, and there's no pg_standby process running at the moment
to receive it? Seems like the signaler needs to loop until it has
successfully delivered the signal to a pg_standby process, which seems
pretty ugly.
Given all the recent trouble with signals, and the fact that it's
undocumented, perhaps we should just rip out the signaling support from
pg_standby.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Fri, 2009-03-27 at 13:19 +0100, Guillaume Smet wrote:
On Fri, Mar 27, 2009 at 12:56 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
Simon Riggs wrote:
If we go with this, I would suggest we make *neither* the default by
removing -t, and adopting two new options: something like -f == fast
failover, -p == patient failover.-m smart|fast|immediate :-)
The advantage of having 2 options (or the ability to put a string
value in the trigger file) is that you can choose the behaviour when
you need to trigger it (you just have to use the 2 options with 2
different filenames). I don't think it's the case with your proposal.
Yes, sorry. I meant we should use the naming Peter suggests.
So we would have two triggers, but call them fast and smart, rather than
fast and patient.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
