beta3 & the open items list
It would be nice to get beta3 out the door sooner rather than later,
but I sort of feel like we're not ready yet. In fact, we seem to be a
bit stalled. The open items list currently lists four items.
1. max_standby_delay. Tom has committed to getting this done, but has
been tied up with non-PostgreSQL related work for the last few weeks.
2. infinite repeat of warning message in standby. Heikki changed the
code so this isn't a tight loop any more, which is an improvement, but
we've discussed the fact that retrying forever may not be the best
behavior.
http://archives.postgresql.org/pgsql-hackers/2010-06/msg00806.php
http://archives.postgresql.org/pgsql-hackers/2010-06/msg00838.php
I am not clear, however, on how difficult it is to implement the
proposed behavior, and I'm not sure Heikki's on board with the
proposed change.
3. supply alternate hstore operator for equals-greater in preparation
for later user in function parameter assignment. There's some work
left to be done here but it's pretty minor. Mostly we're arguing
about whether to call the hstore slice operator +> or & or % or %> --
I've written three patches to rename it so far (to three different
alternative names), one of which I committed, and there's still
ongoing discussion as to whether to rename it again and/or remove it.
Aside from that, we need to deal with the singleton-hstore constructor
(text => text); I believe the consensus there is to remove the
operator in favor of the underlying hstore(text, text) function and
backpatch that function name into the back-branches to facilitate
writing hstore code that is portable across major PostgreSQL releases.
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...
It would be nice if we could make a final push to get these issues
resolved and another beta out the door before the end of the month...
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company
On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...
TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
humbly suggest we *not* be pedantic and implement something practical
and less prone to variables outside the control of Pg.
Sincerely,
Joshua D. Drake
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas@gmail.com> wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...
This sounds like a useful feature but I don't see why it's not 9.1
material. The status quo is that the expected usage pattern is manual
failover. As long as the slave responds to manual intervention when in
this state I don't think this is a blocking issue. Monitoring and
automatic failover are clearly things we plan to add features to
handle better in the future.
--
greg
On Sat, Jun 19, 2010 at 2:46 PM, Greg Stark <gsstark@mit.edu> wrote:
On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas@gmail.com> wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...This sounds like a useful feature but I don't see why it's not 9.1
material. The status quo is that the expected usage pattern is manual
failover. As long as the slave responds to manual intervention when in
this state I don't think this is a blocking issue. Monitoring and
automatic failover are clearly things we plan to add features to
handle better in the future.
Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will). You have to manually restart the server or hit
walreceiver with a SIGTERM to get it to start streaming agian. I
guess we could decide we're just not going to deal with that, but it
seems like a fairly large misfeature to me.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company
Robert Haas <robertmhaas@gmail.com> writes:
Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will).
This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.
I think that enabling slave-side TCP keepalives and control of the
keepalive timeout parameters is probably sufficient for 9.0 here.
regards, tom lane
On Saturday 19 June 2010 18:05:34 Joshua D. Drake wrote:
On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
humbly suggest we *not* be pedantic and implement something practical
and less prone to variables outside the control of Pg.
And has the huge advantage of being implementable in about 5 lines of C
(setsockopt + error checking). Considering what time in the release cycle this
is...
Andres
On 06/19/2010 09:13 PM, Tom Lane wrote:
Robert Haas<robertmhaas@gmail.com> writes:
Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will).This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.I think that enabling slave-side TCP keepalives and control of the
keepalive timeout parameters is probably sufficient for 9.0 here.
yeah I would agree - we do have tcp keepalive code in the backend for a
while now and adding that to libpq as well just seems like an easy
enough fix at this time in the release cycle.
Stefan
On Jun 19, 2010, at 21:13 , Tom Lane wrote:
Robert Haas <robertmhaas@gmail.com> writes:
Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will).This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.
No, Robert is correct AFAIK. If you're *waiting* for data, TCP generates no traffic (expect with keepalive enabled). From the slave's kernel POV, a dead master is therefore indistinguishable from a inactive master.
Things are different from a sender's POV, though. Since sent data is ACK'ed by the receiving end, the TCP stack can (and does) detect a broken connection.
best regards,
Florian Pflug
On Sat, 2010-06-19 at 14:53 -0400, Robert Haas wrote:
On Sat, Jun 19, 2010 at 2:46 PM, Greg Stark <gsstark@mit.edu> wrote:
On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas@gmail.com> wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...This sounds like a useful feature but I don't see why it's not 9.1
material. The status quo is that the expected usage pattern is manual
failover. As long as the slave responds to manual intervention when in
this state I don't think this is a blocking issue. Monitoring and
automatic failover are clearly things we plan to add features to
handle better in the future.Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will). You have to manually restart the server or hit
walreceiver with a SIGTERM to get it to start streaming agian. I
guess we could decide we're just not going to deal with that, but it
seems like a fairly large misfeature to me.
Are you saying it doesn't respond to a trigger file any any point? That
would be a problem.
Sounds like we should have a pg_restart_walreceiver() function. We
shouldn't be encouraging people to send signals to backends, its too
easy to get wrong.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Training and Services
Florian Pflug <fgp@phlo.org> writes:
On Jun 19, 2010, at 21:13 , Tom Lane wrote:
This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.
No, Robert is correct AFAIK. If you're *waiting* for data, TCP
generates no traffic (expect with keepalive enabled).
Mph. I was thinking that keepalive was on by default with a very long
interval, but I see this isn't so. However, if we enable keepalive,
then it's irrelevant to the point anyway. Nobody's produced any
evidence that keepalive is an unsuitable solution.
regards, tom lane
Show quoted text
On Saturday 19 June 2010 18:05:34 Joshua D. Drake wrote:
On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:
4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
humbly suggest we *not* be pedantic and implement something practical
and less prone to variables outside the control of Pg.Sincerely,
++++ +
Joshua D. Drake
On Jun 20, 2010, at 7:18 , Tom Lane wrote:
Florian Pflug <fgp@phlo.org> writes:
On Jun 19, 2010, at 21:13 , Tom Lane wrote:
This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.No, Robert is correct AFAIK. If you're *waiting* for data, TCP
generates no traffic (expect with keepalive enabled).Mph. I was thinking that keepalive was on by default with a very long
interval, but I see this isn't so. However, if we enable keepalive,
then it's irrelevant to the point anyway. Nobody's produced any
evidence that keepalive is an unsuitable solution.
Yeah, I agree. Just enabling keepalive should suffice for 9.0.
BTW, the postmaster already enables keepalive on incoming connections in StreamConnection() - presumably to prevent crashed clients from occupying a backend process forever. So there's even a clear precedent for doing so, and proof that it doesn't cause any harm.
best regards,
Florian Pflug
Florian Pflug wrote:
On Jun 20, 2010, at 7:18 , Tom Lane wrote:
I was thinking that keepalive was on by default with a very
long interval, but I see this isn't so. However, if we enable
keepalive, then it's irrelevant to the point anyway. Nobody's
produced any evidence that keepalive is an unsuitable solution.Yeah, I agree. Just enabling keepalive should suffice for 9.0.
+1, with configurable timeout; otherwise people will often feel they
need to kill the receiver process to get it to attempt reconnect or
archive search, anyway. Two hours is a long time to block
replication based on a broken connection before attempting to move
on.
-Kevin
Import Notes
Resolved by subject fallback
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
Florian Pflug wrote:
Yeah, I agree. Just enabling keepalive should suffice for 9.0.
+1, with configurable timeout;
Right, of course. That's already in the pending patch isn't it?
regards, tom lane
On Sun, 2010-06-20 at 11:36 -0400, Tom Lane wrote:
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
Florian Pflug wrote:
Yeah, I agree. Just enabling keepalive should suffice for 9.0.
+1, with configurable timeout;
Right, of course. That's already in the pending patch isn't it?
Can someone tell me what we are going to do about firewalls that impose
their own rules outside of the control of the DBA?
I know that keepalive *should* work, however I also know that regardless
of keepalive I often have to restart sessions etc. There are
environments that are outside the control of the user.
Perhaps this has already been solved and I don't know about it. Does the
master<->slave relationship have a built in ping mechanism that is
outside of the TCP protocol?
Sincerely,
Joshua D. Drake
regards, tom lane
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
"Joshua D. Drake" wrote:
Can someone tell me what we are going to do about firewalls that
impose their own rules outside of the control of the DBA?
Has anyone actually seen a firewall configured for something so
stupid as to allow *almost* all the various packets involved in using
a TCP connection, but which suppressed just keepalive packets? That
seems to be what you're suggesting is the risk; it's an outlandish
enough suggestion that I think the burden of proof is on you to show
that it happens often enough to make this a worthless change.
-Kevin
Import Notes
Resolved by subject fallback
On Sun, Jun 20, 2010 at 03:01:04PM -0500, Kevin Grittner wrote:
"Joshua D. Drake" wrote:
Can someone tell me what we are going to do about firewalls that
impose their own rules outside of the control of the DBA?Has anyone actually seen a firewall configured for something so
stupid as to allow *almost* all the various packets involved in using
a TCP connection, but which suppressed just keepalive packets? That
seems to be what you're suggesting is the risk; it's an outlandish
enough suggestion that I think the burden of proof is on you to show
that it happens often enough to make this a worthless change.-Kevin
I have seen this sort of behavior but in every case it has been
the result of a myopic view of firewall/IP tables solutions to
perceived "attacks". While I do agree that having heartbeat
within the replication process it worthwhile, it should definitely
be 9.1 material at best. For 9.0 such ill-behaved environments
will need much more interaction by the DBA with monitoring and
triage of problems as they arrive.
Regards,
Ken
P.S. My favorite example of odd behavior was preemptively dropping
TCP packets in one direction only at a single port. Many, many
odd things happen when the kernel does not know that the packet
would never make it to it destination. Services would sometimes
run for weeks without a problem depending on when the port ended
up being used invariably at night or on the weekend.
On Sun, Jun 20, 2010 at 11:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> writes:
Florian Pflug wrote:
Yeah, I agree. Just enabling keepalive should suffice for 9.0.
+1, with configurable timeout;
Right, of course. That's already in the pending patch isn't it?
Is this sarcasm, or is there a pending patch I'm not aware of?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company
Robert Haas <robertmhaas@gmail.com> writes:
On Sun, Jun 20, 2010 at 11:36 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Right, of course. �That's already in the pending patch isn't it?
Is this sarcasm, or is there a pending patch I'm not aware of?
https://commitfest.postgresql.org/action/patch_view?id=281
regards, tom lane
On Jun 20, 2010, at 22:01 , Kevin Grittner wrote:
"Joshua D. Drake" wrote:
Can someone tell me what we are going to do about firewalls that
impose their own rules outside of the control of the DBA?Has anyone actually seen a firewall configured for something so
stupid as to allow *almost* all the various packets involved in using
a TCP connection, but which suppressed just keepalive packets? That
seems to be what you're suggesting is the risk; it's an outlandish
enough suggestion that I think the burden of proof is on you to show
that it happens often enough to make this a worthless change.
Yeah, especially since there is no such thing as a special "keepalive" packet in TCP. Keepalive simply sends packets with zero bytes of payload every once in a while if the connection is otherwise inactive. If those aren't acknowledged (like every other packet would be) by the peer, the connection is assumed to be broken. On a reasonably active connection, keepalive neither causes additional transmissions, nor altered transmissions.
Keepalive is therefore extremely unlikely to break things - in the very worst case, a (really, really stupid) firewall might decide to drop packets with zero bytes of payload, causing inactive connections to abort after a while. AFAIK walreceiver will simply reconnect in this case.
Plus, the postmaster enables keepalive on all incoming connections *already*, so any problems ought to have caused bugreports about dropped client connections.
best regards,
Florian Pflug
