Planning incompatibilities for Postgres 10.0

On Sat, May 25, 2013 at 4:39 AM, Simon Riggs <simon@2ndquadrant.com> wrote:

There are a number of changes we'd probably like to make to the way
things work in Postgres. This thread is not about discussing what
those are, just to say that requirements exist and have been discussed
in various threads over time.

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

So I'd like to make a formal suggestion of a plan for how we cope with this:

1. Implement online upgrade in 9.4 via the various facilities we have
in-progress. That looks completely possible.

2. Name the next release after that 10.0 (would have been 9.5). We
declare now that
a) 10.0 will support on-line upgrade from 9.4 (only)
b) various major incompatibilities will be introduced in 10.0 - the
change in release number will indicate to everybody that is the case
c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
that we will not be constrained by that

This plan doesn't presume any particular change. Each change would
need to be discussed on a separate thread, with a separate case for
each. All I'm suggesting is that we have a coherent plan for the
timing of such changes, so we can bundle them together into one
release.

By doing this now we give ourselves lots of time to plan changes that
will see us good for another decade. If we don't do this, then we
simply risk losing the iniative by continuing to support legacy
formats and approaches.

Huh. I don't think that bumping the version number to 10.0 vs 9.5 is
justification to introduce breaking changes. In fact, I would rather
see 10.0 be the version where we formally stop doing that. I
understand that some stuff needs to be improved but it often doesn't
seem to be worth the cost in the long run.

merlin

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote:

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

Is this proposal only relaxing the binary upgrade requirement, or would
it also relax other compatibility requirements, such as language and API
compatibility?

We need a couple major drivers of the incompatibility that really show
users some value for going through the upgrade pain. Preferably, at
least one would be a serious performance boost, because the users that
encounter the most logical upgrade pain are also the ones that need a
performance boost the most.

Before we set a specific schedule, I think it would be a good idea to
start prototyping some performance improvements that involve breaking
the data format. Then, depending on how achievable it is, we can plan
for however many more 9.X releases we think we need. That being said, I
agree with you that planning in advance is important here, so that
everyone knows when they need to get format-breaking changes in by.

Regards,
Jeff Davis

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 25 May 2013 18:13, Jeff Davis <pgsql@j-davis.com> wrote:

On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote:

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

Is this proposal only relaxing the binary upgrade requirement, or would
it also relax other compatibility requirements, such as language and API
compatibility?

I'm suggesting that as many as possible changes we would like to make
can happen in one release. This is for the benefit of users, so we
dont make every release a source of incompatibilities.

And that release should be the first one where we have online upgrade
possible, which is one after next.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote:

There are a number of changes we'd probably like to make to the way
things work in Postgres. This thread is not about discussing what
those are, just to say that requirements exist and have been discussed
in various threads over time.

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

So I'd like to make a formal suggestion of a plan for how we cope with this:

1. Implement online upgrade in 9.4 via the various facilities we have
in-progress. That looks completely possible.

2. Name the next release after that 10.0 (would have been 9.5). We
declare now that
a) 10.0 will support on-line upgrade from 9.4 (only)
b) various major incompatibilities will be introduced in 10.0 - the
change in release number will indicate to everybody that is the case
c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
that we will not be constrained by that

Assuming online upgrade is going to require logical replication, you are
also assuming 2x storage as you need to have a second cluster to perform
the upgrade. pg_upgrade would still be needed to upgrade a cluster
in-place.

This sounds like, "I created a new tool which does some of what the old
tool does. Let's break the old tool to allow some unspecified changes I
might want to make." I consider this thread to be not thought-through,
obviously.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 25 May 2013 21:44, Bruce Momjian <bruce@momjian.us> wrote:

On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote:

There are a number of changes we'd probably like to make to the way
things work in Postgres. This thread is not about discussing what
those are, just to say that requirements exist and have been discussed
in various threads over time.

The constraint on such changes is that we've decided that we must have
an upgrade path from release to release.

So I'd like to make a formal suggestion of a plan for how we cope with this:

1. Implement online upgrade in 9.4 via the various facilities we have
in-progress. That looks completely possible.

2. Name the next release after that 10.0 (would have been 9.5). We
declare now that
a) 10.0 will support on-line upgrade from 9.4 (only)
b) various major incompatibilities will be introduced in 10.0 - the
change in release number will indicate to everybody that is the case
c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
that we will not be constrained by that

Assuming online upgrade is going to require logical replication, you are
also assuming 2x storage as you need to have a second cluster to perform
the upgrade.

The people that want online upgrade already have 1+ other systems to
do this with.

pg_upgrade would still be needed to upgrade a cluster
in-place.
This sounds like, "I created a new tool which does some of what the old
tool does. Let's break the old tool to allow some unspecified changes I
might want to make."

I haven't argued against pg_upgrade in general, nor said anything
about breaking it. I proposed that we don't support a pg_upgrade path
between two near-future releases, as a way of introducing
incompatibilities. After that, we would continue to use pg_upgrade for
later releases.

Logical replication is being developed, which gives us a complete code
path for doing what we'd need to do. The most important thing is we
wouldn't need to develop any other code that exists just for upgrade.

Writing special code just for pg_upgrade will take a lot of work.
Running that code would mean pg_upgrade would touch the actual
database, which would be down for a long time while it runs. And if it
hits a bug during or after, you're hosed. So you'd need to take a full
backup before you started the process, probably storing it on disk
somewhere and so you would need x2 disk space with this route also.
Specialised code is less well tested, which means bugs are more likely
to occur and tends to perform more poorly. Not only that, but the
first person to want an incompatibility gets to write all the code
needed and take responsibility for the bugs. I can't comment for
others, but I can say I would not personally choose that route - it
looks both expensive and risky.

I consider this thread to be not thought-through, obviously.

My proposal has had lots of serious consideration, but that is not the
topic of this thread.

The title of the thread is a general one, with a clear objective.

I'm looking for a way forwards that allows us to introduce the changes
that many have proposed and which regrettably result in
incompatibilities. If we have no plan I think its likely it will never
happen and it is currently blocking useful change.

Please explain what you consider to be a better plan, so we can judge
all proposals together.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:

I consider this thread to be not thought-through, obviously.

My proposal has had lots of serious consideration, but that is not the
topic of this thread.

The title of the thread is a general one, with a clear objective.

I'm looking for a way forwards that allows us to introduce the changes
that many have proposed and which regrettably result in
incompatibilities. If we have no plan I think its likely it will never
happen and it is currently blocking useful change.

Please explain what you consider to be a better plan, so we can judge
all proposals together.

I agree with the idea of using logical replication as a way to do
pg_upgrade version-breaking releases. What I don't know is what
incompatible changes are pending that would require this.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote:

On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:

I consider this thread to be not thought-through, obviously.

My proposal has had lots of serious consideration, but that is not the
topic of this thread.

The title of the thread is a general one, with a clear objective.

I'm looking for a way forwards that allows us to introduce the changes
that many have proposed and which regrettably result in
incompatibilities. If we have no plan I think its likely it will never
happen and it is currently blocking useful change.

Please explain what you consider to be a better plan, so we can judge
all proposals together.

I agree with the idea of using logical replication as a way to do
pg_upgrade version-breaking releases. What I don't know is what
incompatible changes are pending that would require this.

Sorry I was unclear. When I said "not thought-through", I meant, you
need to start with the _reason_ we need to break pg_upgrade in an
upcoming version, then we can start to plan how to do it. The logical
replication idea is a good one for getting us through pg_upgrade
version-breaking releases.

I am fine with breaking pg_upgrade, but I just don't see the pending
reason at this point.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 05/26/2013 04:22 PM, Bruce Momjian wrote:

On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote:

On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote:

I consider this thread to be not thought-through, obviously.

My proposal has had lots of serious consideration, but that is not the
topic of this thread.

The title of the thread is a general one, with a clear objective.

I'm looking for a way forwards that allows us to introduce the changes
that many have proposed and which regrettably result in
incompatibilities. If we have no plan I think its likely it will never
happen and it is currently blocking useful change.

Please explain what you consider to be a better plan, so we can judge
all proposals together.

I agree with the idea of using logical replication as a way to do
pg_upgrade version-breaking releases. What I don't know is what
incompatible changes are pending that would require this.

Sorry I was unclear. When I said "not thought-through", I meant, you
need to start with the _reason_ we need to break pg_upgrade in an
upcoming version, then we can start to plan how to do it. The logical
replication idea is a good one for getting us through pg_upgrade
version-breaking releases.

I am fine with breaking pg_upgrade, but I just don't see the pending
reason at this point.

Not sure which ones Simon meant, but at least any new/better
storage manager would seem to me to be requiring
a non-pg_upgrade upgrade path unless we require the storage manager
to also include a parallel implementation of pg_upgrade.

The family of possible storage magers here would include column stores,
distributed / partitioned / replicated memory-only / index-structured / ...
storages which all could have advantages in certain situations and whic
all
need an upgrade path.

While you could do this using sequance of first pg_upgrading and then doing
some internal data migration to new storage manager doing this in one go
would be much smoother.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic O�

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Not sure which ones Simon meant, but at least any new/better
storage manager would seem to me to be requiring
a non-pg_upgrade upgrade path unless we require the storage manager
to also include a parallel implementation of pg_upgrade.

Isn't this a bit of horse-cart inversion here? We just hashed out a
tentative, incomplete pseudo-spec for storage managers *yesterday*. We
don't have a complete spec at this point, let alone a development plan,
and it's entirely possible that we'll be able to implement SMs without
breaking pgupgrade.

It's also not at all clear that we can develop SMs in less than 2 years.
I tend to think it unlikely.

First, let's have a few features for which breaking binary compatibility
is a necessity or a clear benefit. Then we'll schedule when to break them.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Josh Berkus (josh@agliodbs.com) wrote:

and it's entirely possible that we'll be able to implement SMs without
breaking pgupgrade.

I'd certainly hope so.. It's certainly not obvious, to me at least,
why a new SM or supporting any of those features would require
breaking pg_upgrade. Perhaps there's something I'm not seeing there,
but it had better be a *really* good reason..

btw, has anyone posted the SM API proposal..? Unfortunately, I think I
had to leave before that was hashed out..

First, let's have a few features for which breaking binary compatibility
is a necessity or a clear benefit. Then we'll schedule when to break them.

Agreed.

Thanks,

Stephen

Stephen Frost <sfrost@snowman.net> writes:

btw, has anyone posted the SM API proposal..? Unfortunately, I think I
had to leave before that was hashed out..

There isn't one yet. We think we understand where the pain points are,
but there's still a long way to go to have a proposal.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

The assumption that we ought to plan expressly for an incompatibility that
essentially discards pg_upgrade seems premature, particularly in advance of
would-be solutions that, in some cases, mightn't actually work.

If pg_upgrade doesn't work, then, at present, the plausible solutions are
to either dump and restore, which might take way too long, or use one of
the logical replication systems (e.g. - Slony, Londiste, or similar, in the
absence of the would-be built-in logical replication).

Unfortunately, there are significant scenarios where none of these work,
particularly for data warehouse-like systems where the database size is so
large that the users cannot afford the disk space to construct a replica.
It sure seems premature to intentionally leave that set of users out in the
cold.

On 05/25/2013 05:39 PM, Simon Riggs wrote:

2. Name the next release after that 10.0 (would have been 9.5). We
declare now that
a) 10.0 will support on-line upgrade from 9.4 (only)
b) various major incompatibilities will be introduced in 10.0 - the
change in release number will indicate to everybody that is the case
c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so
that we will not be constrained by that

While we're talking about changing things, what about:

- Switching to single-major-version release numbering. The number of
people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
etc with a typical major/minor scheme might be worth considering.

- s/cluster/server/g . Just because "cluster" is historical usage
doesn't make it any less confusing for users.

*dives for asbestos fire suit*

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, May 27, 2013 at 2:01 PM, Craig Ringer <craig@2ndquadrant.com> wrote:

On 05/25/2013 05:39 PM, Simon Riggs wrote:
- Switching to single-major-version release numbering. The number of
people who say "PostgreSQL 9.x" is amazing; even *packagers* get this
wrong and produce "postgresql-9" packages. Witness Amazon Linux's awful
PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0,
etc with a typical major/minor scheme might be worth considering.

In this case you don't even need the 2nd digit...
Btw, -1 for the idea, as it would remove the possibility to tell that a new
major release incrementing the 1st digit of version number brings more
enhancement than normal major releases incrementing the 1st digit. This was
the case for 9.0, helping people in remembering that streaming replication
has been introduced from 9.x series.
--
Michael

On Sun, May 26, 2013 at 09:18:41PM -0400, Stephen Frost wrote:

* Josh Berkus (josh@agliodbs.com) wrote:

and it's entirely possible that we'll be able to implement SMs without
breaking pgupgrade.

I'd certainly hope so.. It's certainly not obvious, to me at least,
why a new SM or supporting any of those features would require
breaking pg_upgrade. Perhaps there's something I'm not seeing there,
but it had better be a *really* good reason..

If I had to _guess_, I would say users who are using the default storage
manager would still be able to use pg_upgrade, and those using
non-default storage managers perhaps can't.

But, again, this is all so hypothetical that it doesn't seem worth
talking about. My big point is that someone came to me at PGCon asking
if I knew anything about why Simon thought we needed to break pg_upgrade
in <2 years, and I said no, so I had go digging into my email to find
out what was going on. Simon has a very visible position in the
community, so when he suggests something, people take it seriously,
which means I have to address it. I would prefer if there was more
thought put into the ideas before they are posted.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

* Bruce Momjian (bruce@momjian.us) wrote:

If I had to _guess_, I would say users who are using the default storage
manager would still be able to use pg_upgrade, and those using
non-default storage managers perhaps can't.

That would make sense.

But, again, this is all so hypothetical that it doesn't seem worth
talking about.

Having a specific list of "these are the things we want to change, and
why, and here is why pg_upgrade can't support it" would be much more
useful to work from, I agree.

That said, many discussions and ideas do get shut down, perhaps too
early, because of pg_upgrade considerations. If we had a plan to have
an incompatible release in the future, those ideas and discussions might
be able to progress to a point where we determine it's worth it to take
the pain of a non-pg_upgrade-supported release. That's a bit of a
stretch, in my view, but I suppose it's possible. Even so though, I
would suggest that we put together a wiki page to list out those items
and encourage people to add to such a list; perhaps having an item on
that list would make discussion about it progress beyond "it breaks
pg_upgrade".

Thanks,

Stephen

On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:

* Bruce Momjian (bruce@momjian.us) wrote:

If I had to _guess_, I would say users who are using the default storage
manager would still be able to use pg_upgrade, and those using
non-default storage managers perhaps can't.

That would make sense.

But, again, this is all so hypothetical that it doesn't seem worth
talking about.

Having a specific list of "these are the things we want to change, and
why, and here is why pg_upgrade can't support it" would be much more
useful to work from, I agree.

That said, many discussions and ideas do get shut down, perhaps too
early, because of pg_upgrade considerations. If we had a plan to have
an incompatible release in the future, those ideas and discussions might
be able to progress to a point where we determine it's worth it to take
the pain of a non-pg_upgrade-supported release. That's a bit of a
stretch, in my view, but I suppose it's possible. Even so though, I
would suggest that we put together a wiki page to list out those items
and encourage people to add to such a list; perhaps having an item on
that list would make discussion about it progress beyond "it breaks
pg_upgrade".

Yes, we should be collecting things we want to do for a pg_upgrade break
so we can see the list all in one place.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 05/26/2013 06:18 PM, Josh Berkus wrote:

Not sure which ones Simon meant, but at least any new/better
storage manager would seem to me to be requiring
a non-pg_upgrade upgrade path unless we require the storage manager
to also include a parallel implementation of pg_upgrade.

Isn't this a bit of horse-cart inversion here? We just hashed out a
tentative, incomplete pseudo-spec for storage managers *yesterday*.

Many people have been *thinking* about pluggable storage /
storage managers for much longer time.

We
don't have a complete spec at this point, let alone a development plan,

I think we will have a development plan *before* complete spec
anyway :)

and it's entirely possible that we'll be able to implement SMs without
breaking pgupgrade.

My point was exactly to not spend majority of new storage manager
discussion on "does it break pg_upgrade", "maybe we can find a way
to do it without breaking pg_upgrade", etc...

It's also not at all clear that we can develop SMs in less than 2 years.
I tend to think it unlikely.

I think the important part of Simons message was not "two years"

First, let's have a few features for which breaking binary compatibility
is a necessity or a clear benefit. Then we'll schedule when to break them.

But rather than "it breaks pg_upgrade" not being a complete stopper for
proposed useful features that might break it.

--
Hannu Krosing
PostgreSQL Consultant
Performance, Scalability and High Availability
2ndQuadrant Nordic OÜ

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian <bruce@momjian.us> writes:

On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote:

That said, many discussions and ideas do get shut down, perhaps too
early, because of pg_upgrade considerations. If we had a plan to have
an incompatible release in the future, those ideas and discussions might
be able to progress to a point where we determine it's worth it to take
the pain of a non-pg_upgrade-supported release. That's a bit of a
stretch, in my view, but I suppose it's possible. Even so though, I
would suggest that we put together a wiki page to list out those items
and encourage people to add to such a list; perhaps having an item on
that list would make discussion about it progress beyond "it breaks
pg_upgrade".

Yes, we should be collecting things we want to do for a pg_upgrade break
so we can see the list all in one place.

Precisely. We've said right along that we reserve the right to have a
non-upgradable disk format change whenever sufficiently many reasons
accumulate to do that. The way to go about that is to collect projects
that need to be kept on hold for such a release --- not to say we're
going to have such a release and then look for reasons.

regards, tom lane

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers