pg_upgrade: downgradebility

On Wed, Sep 20, 2006 at 12:54:14PM +0200, Zdenek Kotala wrote:

My first question is how important is downgrade for You and Your customers?

And second is how to verify that downgrade is possible?

Well, one way to do it is to set up a Slony replica using the older
version of the software. So, if you've upgraded to 8.1.x, you
replicate to an old 8.0.x back end as well. If 8.1 doesn't work for
you, you just MOVE everything back to the 8.0.x back end, and you're
golden.

A

--
Andrew Sullivan | ajs@crankycanuck.ca
Information security isn't a technological problem. It's an economics
problem.
--Bruce Schneier

On Wed, 20 Sep 2006, Andrew Sullivan wrote:

On Wed, Sep 20, 2006 at 12:54:14PM +0200, Zdenek Kotala wrote:

My first question is how important is downgrade for You and Your customers?

And second is how to verify that downgrade is possible?

Well, one way to do it is to set up a Slony replica using the older
version of the software. So, if you've upgraded to 8.1.x, you
replicate to an old 8.0.x back end as well. If 8.1 doesn't work for
you, you just MOVE everything back to the 8.0.x back end, and you're
golden.

Well, I think that people who really want downgrade in such a tool are
those for which slony replication is just not an option. That is, data in
the range of hundreds of gigabytes. Using slony to upgrade is often not
practical either.

I wonder if pg_upgrade could be designed in such a way that upgrade is the
same as downgrade from a development point of view. That is, the tool can
change the system from one binary format to another.

Thanks,

Gavin

Gavin Sherry wrote:

On Wed, 20 Sep 2006, Andrew Sullivan wrote:

On Wed, Sep 20, 2006 at 12:54:14PM +0200, Zdenek Kotala wrote:

My first question is how important is downgrade for You and Your customers?

And second is how to verify that downgrade is possible?

Well, one way to do it is to set up a Slony replica using the older
version of the software. So, if you've upgraded to 8.1.x, you
replicate to an old 8.0.x back end as well. If 8.1 doesn't work for
you, you just MOVE everything back to the 8.0.x back end, and you're
golden.

Well, I think that people who really want downgrade in such a tool are
those for which slony replication is just not an option. That is, data in
the range of hundreds of gigabytes. Using slony to upgrade is often not
practical either

I agree with Gavin, Slony need a lot of extra resources. It is similarly
to use pg_dump for upgrade/downgrade.

I wonder if pg_upgrade could be designed in such a way that upgrade is the
same as downgrade from a development point of view. That is, the tool can
change the system from one binary format to another.

The main problem is that newer format has more functionality that older.
There is not possible perform downgrade if some create store procedure
with SQL command which is not supported in the older version. Downgrade
must check it but is it possible to perform 100% check now (without
postgres code change)?

Zdenek

On Wed, Sep 20, 2006 at 09:28:42PM +1000, Gavin Sherry wrote:

Well, I think that people who really want downgrade in such a tool are
those for which slony replication is just not an option. That is, data in
the range of hundreds of gigabytes. Using slony to upgrade is often not
practical either.

Yes, I figured as much. I was just suggesting that people who
_really_ need the fault tolerance of being able to fall back do have
an option, though it's probably somewhat painful to use. (You can
use Slony for hundreds of gigs, although if the throughput is high
enough, it might be tricky to get started.)

I wonder if pg_upgrade could be designed in such a way that upgrade is the
same as downgrade from a development point of view. That is, the tool can
change the system from one binary format to another.

I guess the question is whether there is any case where you can map
an old version of some feature into some number of more-granular new
versions, but going the other way is impossible. I know that sounds
all hand-wavy. The analogy I'm thinking of is cases I've worked on
where protocols change: the old protocol has one flag that can
happily be mapped into, say, three in the new system. But in the new
system, you can have just one of those flags by itself, and there's
no obvious way to preserve that data when moving back to the old
protocol. If we never have that sort of case with the binary
formats, then what you propose ought to work.

A

--
Andrew Sullivan | ajs@crankycanuck.ca
Unfortunately reformatting the Internet is a little more painful
than reformatting your hard drive when it gets out of whack.
--Scott Morris

Zdenek, Andrew,

Overall, I'd tend to say that downgradability should be for a v2 version of
the tool. It's simply not as important. For one thing, there's always
reverting-to-backup.

However, we *do* need to be able to halt the upgrade and reverse it if it
starts erroring out. So that might end up being the same code.

Supposedly someone from EnterpriseDB is working on this as well; where are
you?

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

On 9/20/06, Josh Berkus <josh@agliodbs.com> wrote:

Supposedly someone from EnterpriseDB is working on this as well;
where are you?

Josh,

We're currently working on the design docs and a beta. Denis, Faiz,
and/or Ahsan will post here when it's posted. You should expect some
activity in this area in the next two weeks or so.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

Jonah,

We're currently working on the design docs and a beta. Denis, Faiz,
and/or Ahsan will post here when it's posted. You should expect some
activity in this area in the next two weeks or so.

So, are you saying that you already have code?

If not, care to open up the process? We have others interested in
contributing.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

On 9/21/06, Josh Berkus <josh@agliodbs.com> wrote:

So, are you saying that you already have code?

Yes.

If not, care to open up the process? We have others interested in
contributing.

We'll be opening it in the next few weeks. A large majority of it is
working but we want to solidify the design and current issues before
releasing it for comment and suggestion.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

"Jonah H. Harris" <jonah.harris@gmail.com> writes:

On 9/21/06, Josh Berkus <josh@agliodbs.com> wrote:

If not, care to open up the process? We have others interested in
contributing.

We'll be opening it in the next few weeks. A large majority of it is
working but we want to solidify the design and current issues before
releasing it for comment and suggestion.

I see more and more people falling into the trap of doing a lot of work
*before* showing their ideas to the community. This is an excellent
plan if what you are hoping to accomplish is to waste a lot of work.
Otherwise, you'll do a lot better to get the community involved sooner.

regards, tom lane

Jonah H. Harris wrote:

On 9/21/06, Josh Berkus <josh@agliodbs.com> wrote:

So, are you saying that you already have code?

Yes.

If not, care to open up the process? We have others interested in
contributing.

We'll be opening it in the next few weeks. A large majority of it is
working but we want to solidify the design and current issues before
releasing it for comment and suggestion.

I don't understand why anybody thinks this makes any sense. You know,
if I could post the Mammoth Replicator code for comment, I'd do it as
soon as possible, design flaws and coding bugs included. It would help
me get early feedback on dumb or bright decisions, so that I don't waste
time pursuing fixes that are not invasive enough, or that are too
invasive.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On 9/21/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I see more and more people falling into the trap of doing a lot of work
*before* showing their ideas to the community. This is an excellent
plan if what you are hoping to accomplish is to waste a lot of work.
Otherwise, you'll do a lot better to get the community involved sooner.

Thanks for your opinion, but I disagree completely.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

On 9/21/06, Alvaro Herrera <alvherre@commandprompt.com> wrote:

You know, if I could post the Mammoth Replicator code for comment,
I'd do it as soon as possible, design flaws and coding bugs included.

It's easy to say something on the contrary when you know you can't
back it up to begin with.

Before everyone grabs their pitchforks, I would like to comment that
this was going to be a strictly commercial product and as such, had
already been through some design. We decided to open source it for
everyone and get community involvement, so excuse us if we're going to
spend just a little more time on it.

I don't think I've seen anyone actually propose a design and state
they were going to work on it (at least until 8.3)... so it's funny
that I'm trying to follow the new, "keep the community informed"
policy and yet everyone is still not happy enough.

In short, our design is based on many of the pg_upgrade concepts but
rewritten in C... if you have any blatant faults in pg_upgrade, that's
the best place to start discussion.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

Jonah H. Harris wrote:

On 9/21/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I see more and more people falling into the trap of doing a lot of work
*before* showing their ideas to the community. This is an excellent
plan if what you are hoping to accomplish is to waste a lot of work.
Otherwise, you'll do a lot better to get the community involved sooner.

Thanks for your opinion, but I disagree completely.

History in this community has basically shown that although you disagree
you are completely wrong. Good luck with your project.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Jonah,

I don't think I've seen anyone actually propose a design and state
they were going to work on it (at least until 8.3)... so it's funny
that I'm trying to follow the new, "keep the community informed"
policy and yet everyone is still not happy enough.

Ooops, didn't mean to start a which-hunt. To be fair, I e-mailed Denis
about the project before it came up here, and he didn't respond (still
hasn't).

The current pgFoundry project is fine if this is going to be some
proprietary code which EDB is going to drop on the community. However,
that runs a significant risk of major design issues which will cause the
project to never be used, and to be replaced with something else. Even
the name of the pgFoundry project, as proposed, pretty much prohibits
serious contributions from anyone else: it's "EnterpriseDB Migrator",
not "pg_upgrade".

So I think the thing to decide is whether this is EDB freeware or a real
community OSS effort. I think that we'll be happy to have either, but
right now it's in some grey area.

--Josh Berkus

On 9/21/06, Josh Berkus <josh@agliodbs.com> wrote:

Ooops, didn't mean to start a which-hunt.

Not you're fault :)

So I think the thing to decide is whether this is EDB freeware or a real
community OSS effort. I think that we'll be happy to have either, but
right now it's in some grey area.

We certainly want it to be a community effort, we're just cleaning up
the design right now. The code itself will be posted as-is for anyone
to break, patch, and enhance.

Even if the design is the only thing the community latches onto,
improves, and drives into a new working pg_upgrade, that would be a
good thing. We're just asking for a little patience.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

We would help if we do it in an open pgfoundry project.

Bizgres?

- Luke

Msg is shrt cuz m on ma treo

-----Original Message-----
From: Josh Berkus [mailto:josh@agliodbs.com]
Sent: Thursday, September 21, 2006 11:08 PM Eastern Standard Time
To: Jonah H. Harris
Cc: pgsql-hackers@postgresql.org; Andrew Sullivan
Subject: Re: [HACKERS] pg_upgrade: downgradebility

Jonah,

I don't think I've seen anyone actually propose a design and state
they were going to work on it (at least until 8.3)... so it's funny
that I'm trying to follow the new, "keep the community informed"
policy and yet everyone is still not happy enough.

Ooops, didn't mean to start a which-hunt. To be fair, I e-mailed Denis
about the project before it came up here, and he didn't respond (still
hasn't).

The current pgFoundry project is fine if this is going to be some
proprietary code which EDB is going to drop on the community. However,
that runs a significant risk of major design issues which will cause the
project to never be used, and to be replaced with something else. Even
the name of the pgFoundry project, as proposed, pretty much prohibits
serious contributions from anyone else: it's "EnterpriseDB Migrator",
not "pg_upgrade".

So I think the thing to decide is whether this is EDB freeware or a real
community OSS effort. I think that we'll be happy to have either, but
right now it's in some grey area.

--Josh Berkus

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postgresql.org so that your
message can get through to the mailing list cleanly

Jonah,

We certainly want it to be a community effort, we're just cleaning up
the design right now. The code itself will be posted as-is for anyone
to break, patch, and enhance.

Even if the design is the only thing the community latches onto,
improves, and drives into a new working pg_upgrade, that would be a
good thing. We're just asking for a little patience.

OK ... can we re-name the project, please, though? It's going to be
hard for employees of other companies to contribute to the "EnterpriseDB
Migrator". Maybe "PGMigrator" with EnterpriseDB leading off the
discription?

--Josh Berkus

On 9/21/06, Josh Berkus <josh@agliodbs.com> wrote:

OK ... can we re-name the project, please, though? It's going to be
hard for employees of other companies to contribute to the "EnterpriseDB
Migrator". Maybe "PGMigrator" with EnterpriseDB leading off the
discription?

I can't personally make that call but I'll discuss it with everyone
tomorrow and get back to you.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

"Jonah H. Harris" <jonah.harris@gmail.com> writes:

In short, our design is based on many of the pg_upgrade concepts but
rewritten in C... if you have any blatant faults in pg_upgrade, that's
the best place to start discussion.

Well, actually, there's a reason why pg_upgrade isn't in the CVS tree
anymore ... it never quite worked and for many releases has been in
absolutely no danger of working. So you've not exactly allayed my
concerns by stating the above. I suggest again that you ought to be
running your basic design past the list sooner rather than later.
There are a lot of smart people on this list --- they can help you
not make mistakes.

regards, tom lane

On 9/22/06, Tom Lane <tgl@sss.pgh.pa.us> wrote:

I suggest again that you ought to be running your basic design
past the list sooner rather than later.

We will be posting it as soon as we can.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris@enterprisedb.com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

Jonah,

It's been two weeks, and I haven't seen anything, either here or on
pg_foundry. Is this project derailed?

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco