alpha3 release schedule?
Do people want more time to play with hot standby? Otherwise alpha3
should go out on Monday or Tuesday.
Do people want more time to play with hot standby? Otherwise alpha3
should go out on Monday or Tuesday.
Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.
This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.
If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.
regards,
--
Hiroyuki YAMADA
Kokolink Corporation
yamada@kokolink.net
On Sat, Dec 19, 2009 at 7:20 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
Do people want more time to play with hot standby? Otherwise alpha3
should go out on Monday or Tuesday.
I think we should try to wrap it promptly. It's true that Hot Standby
almost certainly has bugs and/or annoying limitations, as one would
expect with a feature of this magnitude, but I think we'll get a
better idea what they are and which ones are the most important by
getting something out there for people to test. AIUI, the reason why
Simon has been busting ass to get this committed is precisely so that
it could go into alpha3 and get more testing, and speaking in my
capacity as a guy who is anal about the schedule, I couldn't be
happier about that! Postponing alpha3 would seem to defeat the purpose
of all that hard work.
...Robert
Hiroyuki Yamada <yamada@kokolink.net> writes:
Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.
This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.
If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.
Seems like something we should fix ASAP, but I do not see why it need
hold up an alpha release. Alpha releases are expected to have bugs,
and this one doesn't look like it would stop people from finding
other bugs.
regards, tom lane
Tom Lane wrote:
Hiroyuki Yamada <yamada@kokolink.net> writes:
Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.Seems like something we should fix ASAP, but I do not see why it need
hold up an alpha release. Alpha releases are expected to have bugs,
and this one doesn't look like it would stop people from finding
other bugs.
yeah afaik alpha tarballs are a forma of a checkpoint at the end of a
commitfest to get people a reasonable testing target. Every feature (not
only HS) deserves getting serious testing so I vote for getting alpha3
out as soon as possible.
Stefan
Hiroyuki Yamada <yamada@kokolink.net> writes:
Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.Seems like something we should fix ASAP, but I do not see why it need
hold up an alpha release. Alpha releases are expected to have bugs,
and this one doesn't look like it would stop people from finding
other bugs.
At the beginning of this commit fest, Heikki said in
http://archives.postgresql.org/pgsql-hackers/2009-11/msg00914.php
Of course there should be several phases! We've *already* punted a lot
of stuff from this first increment we're currently working on. The
criteria for getting this first phase committed is: could we release
with no further changes?
And other patches seem to be checked with similar criteria, as long as
I read mails in this list. So I wanted to know whether the problem is
must-fix, and if it is, why the criteria has been changed during the
commit fest.
Anyway, thanks for answering my question.
regards,
--
Hiroyuki YAMADA
Kokolink Corporation
yamada@kokolink.net
On Sat, 2009-12-19 at 18:12 +0100, Stefan Kaltenbrunner wrote:
Seems like something we should fix ASAP, but I do not see why it
need
hold up an alpha release. Alpha releases are expected to have bugs,
and this one doesn't look like it would stop people from finding
other bugs.yeah afaik alpha tarballs are a forma of a checkpoint at the end of a
commitfest to get people a reasonable testing target. Every feature
(not
only HS) deserves getting serious testing so I vote for getting
alpha3
out as soon as possible.
+1 for both.
--
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz
Hiroyuki Yamada wrote:
Hiroyuki Yamada <yamada@kokolink.net> writes:
Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.
This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.
If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.Seems like something we should fix ASAP, but I do not see why it need
hold up an alpha release. Alpha releases are expected to have bugs,
and this one doesn't look like it would stop people from finding
other bugs.At the beginning of this commit fest, Heikki said in
http://archives.postgresql.org/pgsql-hackers/2009-11/msg00914.phpOf course there should be several phases! We've *already* punted a lot
of stuff from this first increment we're currently working on. The
criteria for getting this first phase committed is: could we release
with no further changes?And other patches seem to be checked with similar criteria, as long as
I read mails in this list. So I wanted to know whether the problem is
must-fix, and if it is, why the criteria has been changed during the
commit fest.
Well, that was the criteria I used to decide whether to commit or not.
Not everyone agreed to begin with, and the reason I used that criteria
was a selfish one: I didn't want to be forced to fix loose ends after
the commitfest myself. The big reason for that was that I didn't know
how much time I would have for that. I have no complaints about Simon's
commit. Knowing that I'm not on the hook to close the loose ends, I'm
very happy that it's finally in. (That doesn't mean that I'll stop
paying attention to this patch; I will do as much as I have time to.)
Regarding the bugs you found, I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category. I think they need to be fixed before final release, but
there's no need to delay the alpha release for them.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
Well, that was the criteria I used to decide whether to commit or not.
Not everyone agreed to begin with, and the reason I used that criteria
was a selfish one: I didn't want to be forced to fix loose ends after
the commitfest myself. The big reason for that was that I didn't know
how much time I would have for that. I have no complaints about Simon's
commit. Knowing that I'm not on the hook to close the loose ends, I'm
very happy that it's finally in. (That doesn't mean that I'll stop
paying attention to this patch; I will do as much as I have time to.)Regarding the bugs you found, I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category. I think they need to be fixed before final release, but
there's no need to delay the alpha release for them.
I never think it's selfish. But I see. Thanks for your kind reply.
regards,
--
Hiroyuki YAMADA
Kokolink Corporation
yamada@kokolink.net
On Sat, 2009-12-19 at 20:59 +0200, Heikki Linnakangas wrote:
Well, that was the criteria I used to decide whether to commit or not.
Not everyone agreed to begin with, and the reason I used that criteria
was a selfish one: I didn't want to be forced to fix loose ends after
the commitfest myself. The big reason for that was that I didn't know
how much time I would have for that. I have no complaints about Simon's
commit. Knowing that I'm not on the hook to close the loose ends, I'm
very happy that it's finally in. (That doesn't mean that I'll stop
paying attention to this patch; I will do as much as I have time to.)Regarding the bugs you found, I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category. I think they need to be fixed before final release, but
there's no need to delay the alpha release for them.
Hmmm, well, if you are still paying attention you'll know that neither
of those issues are bugs in the code that was committed. One of them has
been fixed and the deadlock problem has a workaround applied.
That workaround, err... works, but I accept its not ideal. But then a
few things are not ideal and it does seem unlikely that every one of
them will have a perfect fix in the next two months.
I'll change the TODO page.
--
Simon Riggs www.2ndQuadrant.com
On Sat, 2009-12-19 at 14:20 +0200, Peter Eisentraut wrote:
Do people want more time to play with hot standby? Otherwise alpha3
should go out on Monday or Tuesday.
No thanks. There were no known bugs in the code I committed, excepting
the need to address VACUUM FULL. That will take longer than two days and
isn't sufficient reason to halt Alpha3, IMHO.
If others wish to revoke, then maybe we should consider a specific Alpha
version just for Hot Standby.
--
Simon Riggs www.2ndQuadrant.com
On Sat, 2009-12-19 at 20:59 +0200, Heikki Linnakangas wrote:
I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category.
I notice you also re-arranged other items on there, specifically the
notion that starting from a shutdown checkpoint is somehow important.
It's definitely not any kind of bug.
We've discussed this on-list and I've requested that you justify this.
So far, nothing you've said on that issue has been at all convincing for
me or others. The topic is already mentioned on the HS todo, since if
one person requests something we should track that, just in case others
eventually agree. But having said that, it clearly isn't a priority, so
rearranging the item like that was not appropriate, unless you were
thinking of doing it yourself, though that wasn't marked.
--
Simon Riggs www.2ndQuadrant.com
On Sat, 2009-12-19 at 23:22 +0900, Hiroyuki Yamada wrote:
Do people want more time to play with hot standby? Otherwise alpha3
should go out on Monday or Tuesday.Well, I want to know whether the problem I refered to
in http://archives.postgresql.org/pgsql-hackers/2009-12/msg01641.php
is must-fix or not.This problem is a corollary of the deadlock problem. This is less catstrophic
but more likely to happen.If you leave this problem, for example, any long-running transactions,
holding any cursors in whatever tables, have a possibility of freezing
whole recovery work in HotStandby node until the transaction commit.
You seem very insistent on bringing up problems just before release.
Almost as if you have a reason to back some other technology other than
this one.
The problem you mention here has been documented and very accessible for
months and not a single person mentioned it up to now. What's more, the
equivalent problem happens in the latest production version of Postgres
- users can delay VACUUM endlessly in just the same way, yet I've not
seen this raised as an issue in many years of using Postgres. Similarly,
there are some ways that Postgres can deadlock that it need not, yet
those negative behaviours are accepted and nobody is rushing to fix
them, nor demanding that they should be. Few things are theoretically
perfect on their first release.
--
Simon Riggs www.2ndQuadrant.com
On Sun, Dec 20, 2009 at 3:42 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
On Sat, 2009-12-19 at 20:59 +0200, Heikki Linnakangas wrote:
I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category.I notice you also re-arranged other items on there, specifically the
notion that starting from a shutdown checkpoint is somehow important.
It's definitely not any kind of bug.We've discussed this on-list and I've requested that you justify this.
So far, nothing you've said on that issue has been at all convincing for
me or others. The topic is already mentioned on the HS todo, since if
one person requests something we should track that, just in case others
eventually agree. But having said that, it clearly isn't a priority, so
rearranging the item like that was not appropriate, unless you were
thinking of doing it yourself, though that wasn't marked.
This doesn't match my recollection of the previous discussion on this
topic. I am not sure that I'd call it a bug, but I'd definitely like
to see it fixed, and I think I mentioned that previously, though I
don't have the email in front ATM. I am also not aware that anyone
other than yourself has opined that we should not worry about fixing
it, although I might be wrong about that too. At any rate, "clearly
not a priority" seems like an overstatement relative to my memory of
that conversation.
...Robert
Simon Riggs wrote:
On Sat, 2009-12-19 at 20:59 +0200, Heikki Linnakangas wrote:
I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category.I notice you also re-arranged other items on there, specifically the
notion that starting from a shutdown checkpoint is somehow important.
I didn't rearrange anything. I added that item because it was missing.
Yes, it is important in my opinion.
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
The problem you mention here has been documented and very accessible for
months and not a single person mentioned it up to now. What's more, the
equivalent problem happens in the latest production version of Postgres
- users can delay VACUUM endlessly in just the same way, yet I've not
seen this raised as an issue in many years of using Postgres. Similarly,
there are some ways that Postgres can deadlock that it need not, yet
those negative behaviours are accepted and nobody is rushing to fix
them, nor demanding that they should be. Few things are theoretically
perfect on their first release.
Sorry for annoying you, at the very first.
Well, this is certainly a well-known problem, but the cursor example
(or deadlock example) reveals that the problem is more severe than
it was considered before, I guess.
Following comments in backup.sgml(which are now replaced by the deadlock example)
Waits for buffer cleanup locks do not currently result in query
cancellation. Long waits are uncommon, though can happen in some cases
with long running nested loop joins.
...refered only to the example where startup process should wait
until the end of one query. And long waits are assumed to be uncommon.
The cursor example shows, however, the waits can be as long as one
transaction, and occur in usual use case. FYI, I wrote a typical freeze
scenario in the mail posted in the original deadlock example thread.
Then the startup process may have to wait until the end of transaction,
and we can not expect when the pin-holder transaction ends.
Also, you mentioned the VACCUM case of the production version, but following
two problems have different impacts.
* One VACUUM process freezes until the end of a certain transaction.
* Startup process(and whole recovery work) freezes until the end of
a certain transaction.
The startup process is the last process to freeze. So I guess
this problem may become must-fix.
Anyway, the patch are committed and alpha 3 are to be released.
Do you think this problem is must-fix for the final release ?
regards,
--
Hiroyuki YAMADA
Kokolink Corporation
yamada@kokolink.net
On Mon, 2009-12-21 at 18:42 +0900, Hiroyuki Yamada wrote:
Do you think this problem is must-fix for the final release ?
We should be clear that this is a behaviour I told you about, not a
shock discovery by yourself. There is no permanent freeze, just a wait,
from which the Startup process wakes up at the appropriate time. There
is no crash or hang as is usually implied by the word freeze.
It remains to be seen whether this is a priority for usability
enhancement in this release. There are other issues as well and it is
doubtful that every user will be fully happy with the functionality in
this release. I will work on things in the order in which I understand
them to be important for the majority, given my time and budget
constraints and the resolvability of the issues.
When you report bugs, I say thanks. When you start agitating about
already-documented restrictions and I see which other software you
promote, I think you may have other motives. Regrettably that reduces
the weight I give your claims, in relation to other potential users.
If you genuinely care about this topic then I hope and expect that you
would start thinking about improvements, or even writing some.
I am already in touch with many potential users and will be engaging
more widely to understand users's reactions from the Alpha release.
--
Simon Riggs www.2ndQuadrant.com
On Sun, 2009-12-20 at 19:11 -0500, Robert Haas wrote:
On Sun, Dec 20, 2009 at 3:42 PM, Simon Riggs <simon@2ndquadrant.com> wrote:
On Sat, 2009-12-19 at 20:59 +0200, Heikki Linnakangas wrote:
I put them on the TODO list at
https://wiki.postgresql.org/wiki/Hot_Standby_TODO, under the must-fix
category.I notice you also re-arranged other items on there, specifically the
notion that starting from a shutdown checkpoint is somehow important.
It's definitely not any kind of bug.We've discussed this on-list and I've requested that you justify this.
So far, nothing you've said on that issue has been at all convincing for
me or others. The topic is already mentioned on the HS todo, since if
one person requests something we should track that, just in case others
eventually agree. But having said that, it clearly isn't a priority, so
rearranging the item like that was not appropriate, unless you were
thinking of doing it yourself, though that wasn't marked.This doesn't match my recollection of the previous discussion on this
topic. I am not sure that I'd call it a bug, but I'd definitely like
to see it fixed, and I think I mentioned that previously, though I
don't have the email in front ATM. I am also not aware that anyone
other than yourself has opined that we should not worry about fixing
it, although I might be wrong about that too. At any rate, "clearly
not a priority" seems like an overstatement relative to my memory of
that conversation.
Please check the thread then. Nobody but me has "opined that we should
not worry about fixing it", but then nobody else other than Heikki has
suggested it is even a feature worthy of inclusion, ever. One person
agreed with my position, nobody has spoken in favour of Heikki's
position. However, I had already included the feature on the todo; it
was further down the todo before a second copy was added, second copy
now removed.
If you are saying being able to start Hot Standby from a shutdown
checkpoint is an important feature for you, then say so, and why.
Please also be careful that you don't mix this up with other
improvements, nor say "they all need fixing". This isn't a general
discussion on those points. There are other important things.
--
Simon Riggs www.2ndQuadrant.com
On 22.12.09 9:34 , Simon Riggs wrote:
If you are saying being able to start Hot Standby from a shutdown
checkpoint is an important feature for you, then say so, and why.
I think it's not so much an important feature but more the removal of a
footgun.
Image a reporting database where all transactions but a few daily bulk
imports are read-only. To spread the load, you do your bulk loads on the
master, but run the reporting queries against a read-only HS slave. Now
you take the master down for maintenance. Since all clients but the bulk
loader use the slave already, and since the bulk loads can be deferred
until after the maintenance window closes again, you don't actually do a
fail-over.
Now you're already pointing at your foot with the gun. All it takes to
ruin your day is *some* reason for the slave to restart. Maybe due to a
junior DBA's typo, or maybe due to a bug in postgres. Anway, once the
slave is down, it won't come up until you manage to get the master up
and running again. And this limitation is pretty surprising, since one
would assume that if the slave survives a *crash* of the master, it'd
certainly survive a simple *shutdown*.
best regards,
Florian Pflug
On Tue, Dec 22, 2009 at 8:34 AM, Simon Riggs <simon@2ndquadrant.com> wrote:
If you are saying being able to start Hot Standby from a shutdown
checkpoint is an important feature for you, then say so, and why.
Can you explain the consequences of missing this? It sounds to me like
if I lose my master and it happened to be while it was shut down for
whatever reason then I'll be stuck and won't be able to use my
standby. If that's true it seems like it's a major problem. Or does it
just mean I would have to follow a different procedure when failing
over?
I'm not sure if it's relevant but one thing to realize is that a lot
of MySQL people are used to doing failovers to do regular maintenance
tasks like creating indexes or making schema changes. Besides,a lot of
sites build in regular failovers to ensure that their failover
procedure works. In both cases they usually want to do a clean shut
down of the master to ensure they don't lose any transactions during
the failover.
--
greg
