pg_upgrade project status

Zdenek Kotala wrote:

The current project is not in good shape. Feature freeze is coming and
nothing is committed. Currently there are three patches in the game:

[...]

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

And what is deadline for it? We can delivery it later with 8.4.1, but
best time for testing is during betas and RC period.

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

cheers

andrew

Andrew Dunstan wrote:

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a pretty
onerous requireemnt for Windows in particular. We do use perl in the
build scripts, but that's only required if you want to compile from source.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:

Andrew Dunstan wrote:

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a pretty
onerous requireemnt for Windows in particular. We do use perl in the
build scripts, but that's only required if you want to compile from
source.

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

Much more reasonable than Korn shell in any case (or any shell for that
matter; I think anything is going to be more of a potentially painful
platform dependency than Perl).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

There are installers for it, but given that we made a point of porting
everything to C to avoid using any scripting languages on end-user
machines when we ported to Windows, it seems strange to relax that
'policy' now for convenience.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Tue, Jan 27, 2009 at 11:39:50AM -0300, Alvaro Herrera wrote:

Heikki Linnakangas wrote:

Andrew Dunstan wrote:

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a pretty
onerous requireemnt for Windows in particular. We do use perl in the
build scripts, but that's only required if you want to compile from
source.

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

Much more reasonable than Korn shell in any case (or any shell for that
matter; I think anything is going to be more of a potentially painful
platform dependency than Perl).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

+1

I agree with Alvaro. Perl is a breeze to install on Windows with
Activestate and that using shell code to perform this task adds a
huge platform dependency to the code. Perl is a known and well defined
quantity for scripting.

Cheers,
Ken

On Tue, Jan 27, 2009 at 8:44 AM, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote:

The current project is not in good shape. Feature freeze is coming and
nothing is committed. Currently there are three patches in the game:

Correction: feature freeze is long past.

1) Space reservation
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php

This patch is mandatory for page online conversion and MUST TO be part
of postgreSQL 8.4. if not ... then we will be at the beginning next
year.

I sent updated version today.

I thought we pretty much had agreement that space reservation was not
a good solution to anything, although I admit I'm not quite clear on
what alternative was being proposed.

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

And what is deadline for it? We can delivery it later with 8.4.1, but
best time for testing is during betas and RC period.

Why is the deadline different than anything else? Surely the deadline
is 11/1/2008, and you missed it.

Admittedly, there is some window for reworking existing patches after
the start of the commitfest, but this patch wasn't even added to the
CommitFest wiki until December 5th, after being sent to the list the
previous day. That's not close, and there's been little discussion of
it on the mailing list since then, and given that it's written in ksh,
it's clearly going to require a complete rewrite to be committable.

Three months after the CommitFest started is not the right time to
start a complete rewrite of a feature that wasn't even on time in the
first place. There is nothing whatever to prevent you from releasing
this on pgfoundry, but the idea that we should spend time on this
rather than the half a dozen (or more) patches that have had far more
work and are probably far closer to being committable strikes me as
100% wrong.

3) preupgrade script
http://archives.postgresql.org/pgsql-hackers/2008-12/msg01273.php

I sent WIP version of this script, which shows how reservation space
feature will be used. This part is not important now. It will be
important for 8.4->8.5 upgrade and cannot be finished until 8.5beta.

...Robert

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade.

What's the reason this script uses a postmaster? It seems it would be
easier to control if you used a standalone backend (--single) for each
time you are piping stuff to psql. That would reduce the need to
configure authentication, hostnames, etc etc.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 1/27/09, Zdenek Kotala <Zdenek.Kotala@sun.com> wrote:

This patch is mandatory for page online conversion and MUST TO be part
of postgreSQL 8.4. if not ... then we will be at the beginning next
year.

Just to clarify, does that mean that your patch has to be in for there
to be any chance of in-place upgrade 8.4->8.5?

merlin

Heikki Linnakangas wrote:

Andrew Dunstan wrote:

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

I have had a very brief look at this. Translation to perl doesn't
look difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a
pretty onerous requireemnt for Windows in particular. We do use perl
in the build scripts, but that's only required if you want to compile
from source.

Well, from that POV the only portable thing is to translate it into C.
That's just a whole lot more work (remember initdb?). The perl port for
Windows is easily installable, widely used and well regarded. It doesn't
strike me as too high a price to pay for the ability to do upgrades, but
I'll defer to more Windows-centric commenters.

cheers

andrew

Dave Page wrote:

On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

There are installers for it, but given that we made a point of porting
everything to C to avoid using any scripting languages on end-user
machines when we ported to Windows, it seems strange to relax that
'policy' now for convenience.

If you prefer to not have pg_upgrade at all for 8.4, feel free to
request it to be written in C ... But I'm sure that's not what you
meant.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Dave Page <dpage@pgadmin.org> writes:

On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

There are installers for it, but given that we made a point of porting
everything to C to avoid using any scripting languages on end-user
machines when we ported to Windows, it seems strange to relax that
'policy' now for convenience.

Indeed. We might put up with a perl script for awhile for the sake of
development expediency, but the long-term expectation would have to be
that someone would rewrite it in C. Given that, I wonder whether
there's much point in a rewrite into Perl if we already have a working
shell script. I suppose someone will say "but you'll get no testing
from Windows users then..."

regards, tom lane

Andrew Dunstan wrote:

Heikki Linnakangas wrote:

Andrew Dunstan wrote:

Zdenek Kotala wrote:

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

I have had a very brief look at this. Translation to perl doesn't
look difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a
pretty onerous requireemnt for Windows in particular. We do use perl
in the build scripts, but that's only required if you want to compile
from source.

Well, from that POV the only portable thing is to translate it into C.
That's just a whole lot more work (remember initdb?). The perl port for
Windows is easily installable, widely used and well regarded. It doesn't
strike me as too high a price to pay for the ability to do upgrades, but
I'll defer to more Windows-centric commenters.

Either way, there's no point to discuss that in detail until there
actually is a working implementation out there... perl will do fine
until then. Once we have that, we can discuss if doing it in C will be
worthwhile, or if we're just going to require perl for that one feature.

I have a hard time thinking that we'll have wasted a lot of time on
first doing a perl implementation if we have to rewrite it in C later.
The other way around would be a waste though. The amount of time spent
on the perl implementation I expect to be a *lot* less than the
combination of thinking up the *way* to do it in general and the C
implementation time.

//Magnus

On Tue, Jan 27, 2009 at 2:49 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

Dave Page wrote:

On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
<alvherre@commandprompt.com> wrote:

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

There are installers for it, but given that we made a point of porting
everything to C to avoid using any scripting languages on end-user
machines when we ported to Windows, it seems strange to relax that
'policy' now for convenience.

If you prefer to not have pg_upgrade at all for 8.4, feel free to
request it to be written in C ... But I'm sure that's not what you
meant.

I'd rather it was written in an appropriate language before feature
freeze, not in a language that makes it easier for the author but a
shade harder for thousands of users three months into feature freeze.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

* Magnus Hagander (magnus@hagander.net) wrote:

Either way, there's no point to discuss that in detail until there
actually is a working implementation out there... perl will do fine
until then. Once we have that, we can discuss if doing it in C will be
worthwhile, or if we're just going to require perl for that one feature.

+1

Stephen

* Robert Haas (robertmhaas@gmail.com) wrote:
[pg_upgrade...]

Why is the deadline different than anything else?

err, isn't it because it'd be kind of difficult to do an upgrade script
with large catalog-changing patches outstanding..? I thought some
leeway was given for pg_upgrade specifically due to that, tho perhaps
I'm wrong.

Thanks,

Stephen

On Tue, Jan 27, 2009 at 10:08 AM, Stephen Frost <sfrost@snowman.net> wrote:

* Robert Haas (robertmhaas@gmail.com) wrote:
[pg_upgrade...]

Why is the deadline different than anything else?

err, isn't it because it'd be kind of difficult to do an upgrade script
with large catalog-changing patches outstanding..? I thought some
leeway was given for pg_upgrade specifically due to that, tho perhaps
I'm wrong.

Sure... if this script had been 100% commitable on 11/1 and now needed
to be adjusted, I can't imagine anyone objecting. But the patch
wasn't submitted until 12/4 and still needs a complete rewrite in a
different programming language as of 1/27. Do you think we would be
arguing about whether to accept Hot Standby now if it were written in
ksh? And that was at least submitted on time.

...Robert

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

Much more reasonable than Korn shell in any case (or any shell for that
matter; I think anything is going to be more of a potentially painful
platform dependency than Perl).

May I humbly recommend to rewrite in Python? That should be as
difficult / easy as PERL, AND there is a very robust py2exe
implementation, which allows to create a single .exe file which
contains everything.

Python is present on all Linux, Windows users are totally comfortable
with .exe files.

Harald

--
GHUM Harald Massa
persuadere et programmare
Harald Armin Massa
Spielberger Straße 49
70435 Stuttgart
0173/9409607
no fx, no carrier pigeon
-
EuroPython 2009 will take place in Birmingham - Stay tuned!

Hi

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

Perhaps I can lend you a hand if you need help with this.

--

Med venlig hilsen
Kaare Rasmussen, Jasonic

Jasonic Telefon: +45 3816 2582
Nordre Fasanvej 12
2000 Frederiksberg Email: kaare@jasonic.dk

Harald Armin Massa wrote:

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

Much more reasonable than Korn shell in any case (or any shell for that
matter; I think anything is going to be more of a potentially painful
platform dependency than Perl).

May I humbly recommend to rewrite in Python? That should be as
difficult / easy as PERL, AND there is a very robust py2exe
implementation, which allows to create a single .exe file which
contains everything.

Python is present on all Linux, Windows users are totally comfortable
with .exe files.

No, I don't think so ;-) Without getting into language wars, Perl is
already our de facto cross-platform scripting tool. We don't need to be
adding extra knowledge requirements to the project, nor extra build
requirements (right now, perl is already required for building from
source, or when building with MSVC, or when running a buildfarm animal)

cheers

andrew