Synchronous replication & Hot standby patches
Hi,
Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
Thanks,
Niranjan
K, Niranjan (NSN - IN/Bangalore) wrote:
Hi,
Could you please let me know what are the outstanding features
that are still to be developed in the respective patches?I'am currently referring the wiki: "Todo and Claim" for NTT and
for HotStandby, i see that almost all issues are closed. Are
there any features / refactoring / bugs still need to be fixed.
Heikki is reviewing the hot standby code; we are trying to figure out
how to address the GIN patch, and the SE-Linux patch has gone through
several modifications and is waiting on someone to review it again.
The hash patch also is waiting for performance testing, I think.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Hi,
On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
<niranjan.k@nsn.com> wrote:
Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
At least I'm planning to work on the following two items of Synch Rep for v8.5.
Of course, Synch Rep works fine without these features.
- Address the problem which the signals cannot interrupt poll() and select()
on some platforms.
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00438.php
- Add new feature which transfers all WAL records via the direct connection
between the primary and the standby. In other words, get rid of
file-based log shipping part from the patch.
http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
<niranjan.k@nsn.com> wrote:Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
At least I'm planning to work on the following two items of Synch Rep for v8.5.
Of course, Synch Rep works fine without these features.
- Add new feature which transfers all WAL records via the direct connection
between the primary and the standby. In other words, get rid of
file-based log shipping part from the patch.
http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com
Please bear in mind my strong objection to this. Attempting to transfer
all data via a single connection destroys VLDB usage of this feature. So
for me its just additional code for ease-of-use in the simplest case,
not code replacement.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Simon Riggs wrote:
On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
<niranjan.k@nsn.com> wrote:Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
At least I'm planning to work on the following two items of Synch Rep for v8.5.
Of course, Synch Rep works fine without these features.- Add new feature which transfers all WAL records via the direct connection
between the primary and the standby. In other words, get rid of
file-based log shipping part from the patch.
http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.comPlease bear in mind my strong objection to this. Attempting to transfer
all data via a single connection destroys VLDB usage of this feature. So
for me its just additional code for ease-of-use in the simplest case,
not code replacement.
Presumably we'll keep the capability to restore from a backup and
restore from WAL archive as well, when those are available. Keeping that
capability shouldn't add many lines of code.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
On Tue, 2009-02-24 at 17:36 +0000, Simon Riggs wrote:
On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
<niranjan.k@nsn.com> wrote:Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
At least I'm planning to work on the following two items of Synch Rep for v8.5.
Of course, Synch Rep works fine without these features.- Add new feature which transfers all WAL records via the direct connection
between the primary and the standby. In other words, get rid of
file-based log shipping part from the patch.
http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.comPlease bear in mind my strong objection to this. Attempting to transfer
all data via a single connection destroys VLDB usage of this feature. So
for me its just additional code for ease-of-use in the simplest case,
not code replacement.
Well VLDB is like 2% of what we need. If the above will remove all the
B.S. currently associated with actually doing PITR (rsync, scp, nfs,
pg_standby pick your poison) then I am all for it.
Log shipping should be:
I am master, my slave is here.
I am slave, I understand my master is here.
Here is our mutual authentication love token.
Let congress begin.
Anything more and we are being difficult for the sake of being
difficult.
Joshua D. Drake
--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
On Tue, 2009-02-24 at 18:48 +0000, Simon Riggs wrote:
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.
See Heikki's comment.
Joshua D. Drake
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997
Simon Riggs wrote:
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.
Perhaps so, but I would hope you would support what Heikki and others
have been talking about as an option for replication. The 2% shouldn't
hold back the remaining 98%.
cheers
andrew
Andrew Dunstan <andrew@dunslane.net> wrote:
Simon Riggs wrote:
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to
some users. We shouldn't need to debate the exact percentage of
users that would be affected, or how to count them.Perhaps so, but I would hope you would support what Heikki and
others have been talking about as an option for replication. The 2%
shouldn't hold back the remaining 98%.
Right. I was made a bit nervous by Joshua's comments, but somewhat
reassured by his reference back to Heikki's comments. If we can make
common cases simple to implement, that's great, as long as we don't
lose functionality needed to cover the more complex cases.
Anything more and we are being difficult for the sake of being
difficult.
Even in context, that came off as a bit user-hostile. It probably
wasn't meant that way, but it sounded provocative to me. We do have
better reasons than that for what I mentioned (in simplified form)
here:
http://archives.postgresql.org/pgsql-hackers/2009-01/msg00740.php
If the point was that we should not require anything more for
configuring a common, simple case then I'd agree.
-Kevin
Hi,
On Wed, Feb 25, 2009 at 2:37 AM, Heikki Linnakangas
<heikki.linnakangas@enterprisedb.com> wrote:
Presumably we'll keep the capability to restore from a backup and restore
from WAL archive as well, when those are available. Keeping that capability
shouldn't add many lines of code.
Yes, I assume that only missing WAL files (probably including .backup file)
which are not in archive area of the standby server need to be transferred.
If there are many missing files, we can also copy them via multiple connection
before starting the standby.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Tue, 2009-02-24 at 13:25 -0600, Kevin Grittner wrote:
Right. I was made a bit nervous by Joshua's comments, but somewhat
reassured by his reference back to Heikki's comments. If we can make
common cases simple to implement, that's great, as long as we don't
lose functionality needed to cover the more complex cases.
Right. That is my only point. I am not looking to "dumb down"
postgresql. I am saying that the "default" solution should be that
simple.
Autovacuum is the same way for example. The "default" solution works for
95% of the users out there without having to tweak any knob. The knobs
are there *if* they break out of the 95%.
Anything more and we are being difficult for the sake of being
difficult.Even in context, that came off as a bit user-hostile. It probably
wasn't meant that way, but it sounded provocative to me.
Sorry I am in rare form today. I wasn't trying to be provocative at all
but was trying to be succinctly clear on what my belief about this is.
If the point was that we should not require anything more for
configuring a common, simple case then I'd agree.
Right.
Joshua D. Drake
--
PostgreSQL - XMPP: jdrake@jabber.postgresql.org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997
On Tue, 2009-02-24 at 13:53 -0500, Andrew Dunstan wrote:
Simon Riggs wrote:
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.
Perhaps so, but I would hope you would support what Heikki and others
have been talking about as an option for replication. The 2% shouldn't
hold back the remaining 98%.
So far, everything has been couched in terms of remove the way it is now
and put in its place something "better". Heikki and Josh have said that
or similar, as has Robert Haas on another thread, and Fujii-san
specifically said "get rid of" the existing functionality. I am
completely against the removal of an existing capability that is
critically important to many users.
If we can add new functionality that is a nice-to-have for a large
number of people without removing a feature that is critical to many
users, bring it on. If we can't do that, then I would oppose.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Hi,
On Wed, Feb 25, 2009 at 5:08 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
So far, everything has been couched in terms of remove the way it is now
and put in its place something "better". Heikki and Josh have said that
or similar, as has Robert Haas on another thread, and Fujii-san
specifically said "get rid of" the existing functionality. I am
completely against the removal of an existing capability that is
critically important to many users.
Though my description may be confusing, I don't mean to remove
any existing features unless necessary. I just said get rid of
file-based log shipping part only from my patch.
If we can add new functionality that is a nice-to-have for a large
number of people without removing a feature that is critical to many
users, bring it on. If we can't do that, then I would oppose.
Yes, I also think so.
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Tue, Feb 24, 2009 at 3:08 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
On Tue, 2009-02-24 at 13:53 -0500, Andrew Dunstan wrote:
Simon Riggs wrote:
On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
Well VLDB is like 2% of what we need.
I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.Perhaps so, but I would hope you would support what Heikki and others
have been talking about as an option for replication. The 2% shouldn't
hold back the remaining 98%.So far, everything has been couched in terms of remove the way it is now
and put in its place something "better". Heikki and Josh have said that
or similar, as has Robert Haas on another thread, and Fujii-san
specifically said "get rid of" the existing functionality. I am
completely against the removal of an existing capability that is
critically important to many users.
I didn't think I had proposed any such thing, although maybe I'm just
not remembering. I'm pretty confused as to what the current thread is
all about.
It seems to me that in previous discussions of Streaming Replication,
Heikki put forward the proposition that the standby server should be
able to connect to the primary and stream not only newly-generated WAL
but also, if necessary, a base backup. As I recall, he argued that
without this functionality Streaming Replication would be far too
difficult to administer for the majority of users. Assuming I'm
representing his position more or less accurately, I completely agree
with it. I don't object to providing other mechanisms as well, but if
it's not about as simple as pointing the secondary at the primary and
saying "go", it's probably more complicated than I want to mess around
with.
...Robert
Hi,
On Wed, Feb 25, 2009 at 6:52 AM, Robert Haas <robertmhaas@gmail.com> wrote:
It seems to me that in previous discussions of Streaming Replication,
Heikki put forward the proposition that the standby server should be
able to connect to the primary and stream not only newly-generated WAL
but also, if necessary, a base backup. As I recall, he argued that
without this functionality Streaming Replication would be far too
difficult to administer for the majority of users.
The automatic transfer of a base backup is useful, but not essential.
So, I'm not planning to work on it until synch rep patch is committed
at least. IIRC, nobody argues that such a feature is indispensable
also to the first step. We should develop synch rep in stages?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
On Tue, 2009-02-24 at 16:52 -0500, Robert Haas wrote:
I didn't think I had proposed any such thing, although maybe I'm just
not remembering. I'm pretty confused as to what the current thread is
all about.
http://archives.postgresql.org/pgsql-hackers/2009-01/msg00978.php
I don't think anyone who argued in favour of removal of existing system
was aware that we'd lose anything as a result. I think everybody
supports the easier-if-possible sentiment that Heikki was expressing; I
just don't want to let that be seen as agreement to remove, by default,
at a later time.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Bruce Momjian wrote:
K, Niranjan (NSN - IN/Bangalore) wrote:
Hi,
Could you please let me know what are the outstanding features
that are still to be developed in the respective patches?I'am currently referring the wiki: "Todo and Claim" for NTT and
for HotStandby, i see that almost all issues are closed. Are
there any features / refactoring / bugs still need to be fixed.Heikki is reviewing the hot standby code; we are trying to figure out
how to address the GIN patch, and the SE-Linux patch has gone through
several modifications and is waiting on someone to review it again.
I have updated my patch set a few times to fix bugs since I submitted
the revision without row-level facilities (r1522) at this Feb.06.
Was it unconfortable for you? These changes are just only bugfixes,
so it is not necessary whole of patches again.
Or, does the "several modifications" mean the separation of row-level
facilities and so on?
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>
On Tue, Feb 24, 2009 at 5:58 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
On Tue, 2009-02-24 at 16:52 -0500, Robert Haas wrote:
I didn't think I had proposed any such thing, although maybe I'm just
not remembering. I'm pretty confused as to what the current thread is
all about.http://archives.postgresql.org/pgsql-hackers/2009-01/msg00978.php
I still don't see where I suggested removing anything. What Heikki
suggested, and I agreed with, was adding something: integrated base
backup.
I don't think anyone who argued in favour of removal of existing system
was aware that we'd lose anything as a result. I think everybody
supports the easier-if-possible sentiment that Heikki was expressing; I
just don't want to let that be seen as agreement to remove, by default,
at a later time.
I'm still totally unclear as to what you think anyone might, at some
point in the future, propose to remove.
I think the more relevant question right now is whether the work Fujii
Masao is planning to do for 8.5 is reponsive to the following comment
from Heikki:
# IMHO, the synchronous replication isn't in such good shape, I'm
afraid. I've said
# this before, but I'm not happy with the "built from spare parts"
nature of it. You
# shouldn't have to configure an archive, file-based log shipping using rsync or
# whatever, and pg_standby. All that is in addition to the direct
connection between
# master and slave. The slave really should be able to just connect to
the master, and
# download all the WAL it needs directly. That's a huge usability
issue if left as is,
# but requires very large architectural changes to fix.
...Robert
Robert Haas wrote:
I think the more relevant question right now is whether the work Fujii
Masao is planning to do for 8.5 is reponsive to the following comment
from Heikki:# IMHO, the synchronous replication isn't in such good shape, I'm
afraid. I've said
# this before, but I'm not happy with the "built from spare parts"
nature of it. You
# shouldn't have to configure an archive, file-based log shipping using rsync or
# whatever, and pg_standby. All that is in addition to the direct
connection between
# master and slave. The slave really should be able to just connect to
the master, and
# download all the WAL it needs directly. That's a huge usability
issue if left as is,
# but requires very large architectural changes to fix.
I believe so, see second bullet point in:
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
