libpq API incompatibility between 7.4 and 8.0

Peter Eisentraut <peter_e@gmx.net> writes:

Martin Pitt has detected that the libpq API has changed incompatibly
between 7.4 and 8.0. This has the effect, for example, that 7.4's psql
cannot run with 8.0's libpq.

[ shrug... ] I don't think we've ever guaranteed that anyway. I will
resist putting get_progname back into libpq, because it should never
have been there, at least not with that name: it's an undocumented
infringement on application namespace.

regards, tom lane

Hi!

(sorry for the additional addresses; I'm not subscribed to -hackers,
so my mail will last a while until it arrives there).

Tom Lane [2005-02-02 11:07 -0500]:

Peter Eisentraut <peter_e@gmx.net> writes:

Martin Pitt has detected that the libpq API has changed incompatibly
between 7.4 and 8.0. This has the effect, for example, that 7.4's psql
cannot run with 8.0's libpq.

[ shrug... ] I don't think we've ever guaranteed that anyway.

If you don't guarantee a backwards compatible API, then you should
give libpq a new SONAME. By keeping the SONAME "3" you did promise
backwards compatibility.

The problem with this issue is that it is a real pain for distributors
to handle broken SONAMEs. If every distributor invents his own one,
then there will be conflicts at some point. This gets worse if you do
release a new libpq SONAME later, which might conflict with any fake
SONAME a distributor might have invented.

I will resist putting get_progname back into libpq, because it
should never have been there, at least not with that name: it's an
undocumented infringement on application namespace.

I understand such concerns, but you cannot undo the history. libpq3
is out there and installed on maybe hundreds of thousands of machines,
and your own psql frontend uses it.

The next best option would be to fix the 7.4 version of psql to not
use this symbol any more. However, this is still a pain since then you
cannot upgrade from earlier versions to 8.0 any more.

What would you propose as a solution? There can only be one
/usr/lib/libpq.so.3.

Thanks,

Martin

--
Martin Pitt http://www.piware.de
Ubuntu Developer http://www.ubuntulinux.org
Debian GNU/Linux Developer http://www.debian.org

Martin Pitt <mpitt@debian.org> writes:

What would you propose as a solution?

Do nothing. The problem you are raising isn't very serious since
RPM-style installations don't support concurrent installation of
multiple PG versions anyway. That being the case, it doesn't really
matter whether 8.0 psql can use a 7.4 library or vice versa.

To do otherwise would essentially amount to deciding that get_progname
is part of the exported API of libpq forevermore. That's not something
I'm willing to buy into. It was a mistake that it was done that way in
7.4, and I want to rectify that mistake before it gets any more
entrenched.

regards, tom lane

Tom Lane wrote:

Martin Pitt <mpitt@debian.org> writes:

What would you propose as a solution?

Do nothing. The problem you are raising isn't very serious since
RPM-style installations don't support concurrent installation of
multiple PG versions anyway. That being the case, it doesn't really
matter whether 8.0 psql can use a 7.4 library or vice versa.

Partial upgrades is the key word.

To do otherwise would essentially amount to deciding that
get_progname is part of the exported API of libpq forevermore.
That's not something I'm willing to buy into. It was a mistake that
it was done that way in 7.4, and I want to rectify that mistake
before it gets any more entrenched.

We can rectify the mistake, but then we need to change the SONAME.
That's what it's for.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e@gmx.net> writes:

We can rectify the mistake, but then we need to change the SONAME.
That's what it's for.

Well, if you just want to bump libpq's SO_MAJOR_VERSION, I won't object.

This brings up a point that I think has been discussed before: we
operate on the assumption that incrementing SO_MINOR_VERSION is enough
to distinguish different releases of libpq, but in point of fact it
is not. The Linux conventions for library names, for one, essentially
require us to bump SO_MAJOR_VERSION for every release if we want to have
any hope of letting different versions coexist. Perhaps our convention
should be to bump SO_MAJOR_VERSION for each of our major releases and
only use SO_MINOR_VERSION when we change a library in a dot-release.

regards, tom lane

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

We can rectify the mistake, but then we need to change the SONAME.
That's what it's for.

Well, if you just want to bump libpq's SO_MAJOR_VERSION, I won't object.

This brings up a point that I think has been discussed before: we
operate on the assumption that incrementing SO_MINOR_VERSION is enough
to distinguish different releases of libpq, but in point of fact it
is not. The Linux conventions for library names, for one, essentially
require us to bump SO_MAJOR_VERSION for every release if we want to have
any hope of letting different versions coexist. Perhaps our convention
should be to bump SO_MAJOR_VERSION for each of our major releases and
only use SO_MINOR_VERSION when we change a library in a dot-release.

According to our RELEASE_CHANGES documentation:

The major version number should be updated whenever the source of the
library changes to make it binary incompatible. Such changes include,
but are not limited to:

1. Removing a public function or structure (or typedef, enum, ...)

2. Modifying a public functions arguments.

3. Removing a field from a public structure.

so while I don't think we need to update the major number for every
PostgreSQL major release, the removal of prog_name probably required a
major bump.

Oops.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

According to our RELEASE_CHANGES documentation:

The major version number should be updated whenever the source of the
library changes to make it binary incompatible. Such changes include,
but are not limited to:

1. Removing a public function or structure (or typedef, enum, ...)

2. Modifying a public functions arguments.

3. Removing a field from a public structure.

so while I don't think we need to update the major number for every
PostgreSQL major release, the removal of prog_name probably required a
major bump.

Well, the point is that get_progname *isn't* a "public" function.
We never advertised it as a libpq entry point.

What this really brings out to me is that our development process
doesn't impose a very strong boundary between libpq and our bundled
client programs.  If the client programs were enforced to use only the
documented public API of libpq then we'd not be having this discussion
--- but stuff such as libpgport support functions tends to slip by under
the radar.  IIRC we've been bitten in exactly this way at least once
before.  What I'm suggesting is that we just solve the whole class of
problems permanently, by abandoning the assumption that we're going to
guarantee binary compatibility across major releases.  I don't think
that promise is really buying us anything very critical.

If we don't go that way, then we need to have some automatic check that
none of the client programs are using symbols they shouldn't be from
libpq. (Hmm ... will the existence of the Windows port help here?)

regards, tom lane

Tom Lane wrote:

Bruce Momjian <pgman@candle.pha.pa.us> writes:

According to our RELEASE_CHANGES documentation:

The major version number should be updated whenever the source of the
library changes to make it binary incompatible. Such changes include,
but are not limited to:

1. Removing a public function or structure (or typedef, enum, ...)

2. Modifying a public functions arguments.

3. Removing a field from a public structure.

so while I don't think we need to update the major number for every
PostgreSQL major release, the removal of prog_name probably required a
major bump.

Well, the point is that get_progname *isn't* a "public" function.
We never advertised it as a libpq entry point.

What this really brings out to me is that our development process
doesn't impose a very strong boundary between libpq and our bundled
client programs.  If the client programs were enforced to use only the
documented public API of libpq then we'd not be having this discussion
--- but stuff such as libpgport support functions tends to slip by under
the radar.  IIRC we've been bitten in exactly this way at least once
before.  What I'm suggesting is that we just solve the whole class of
problems permanently, by abandoning the assumption that we're going to
guarantee binary compatibility across major releases.  I don't think
that promise is really buying us anything very critical.

If we don't go that way, then we need to have some automatic check that
none of the client programs are using symbols they shouldn't be from
libpq. (Hmm ... will the existence of the Windows port help here?)

Yes, I think Win32 will help as long as we don't let bad stuff get into
libpqddll.def. The only downside I see to bumping the major number each
time is that the major number could get pretty big. Do the dynamic
library systems handle two-digit library version numbers properly?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

The only downside I see to bumping the major
number each time is that the major number could get pretty big. Do
the dynamic library systems handle two-digit library version numbers
properly?

MySQL's client library is at 12, so I don't see a problem.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Hi Tom!

Tom Lane [2005-02-02 12:01 -0500]:

Martin Pitt <mpitt@debian.org> writes:

What would you propose as a solution?

Do nothing.

That's unfortunately not an option.

The problem you are raising isn't very serious since
RPM-style installations don't support concurrent installation of
multiple PG versions anyway. That being the case, it doesn't really
matter whether 8.0 psql can use a 7.4 library or vice versa.

That's exactly the point I want to change for Debian, I work on a
structure which permits to run several clusters of different versions
in parallel (see [1]http://people.debian.org/~mpitt/postgresql-ng.html). We now have a /usr/lib/libpq.so.3 for 7.4, and
we can't have a _second_ incompatible /usr/lib/libpq.so.3 for 8.0.

Sorry, but that's just the way how shared libraries work. SONAMEs are
for API compatibility and it becomes _very_ hard to workaround broken
ones.

To do otherwise would essentially amount to deciding that get_progname
is part of the exported API of libpq forevermore.

That's not true. It must stay part of the exported API for SONAME 3 of
libpq, not for anything else.

That's not something I'm willing to buy into. It was a mistake that
it was done that way in 7.4, and I want to rectify that mistake
before it gets any more entrenched.

Then please release 8.0.2 with a SONAME 4. Would you consider this?

Thanks a lot and have a nice day!

Martin

[1]: http://people.debian.org/~mpitt/postgresql-ng.html

--
Martin Pitt http://www.piware.de
Ubuntu Developer http://www.ubuntulinux.org
Debian GNU/Linux Developer http://www.debian.org

Tom Lane wrote:

Well, if you just want to bump libpq's SO_MAJOR_VERSION, I won't
object.

Yes. Unless someone objects, I will do that for 8.0.* and 8.1.*.

The Linux conventions for library names, for one,
essentially require us to bump SO_MAJOR_VERSION for every release if
we want to have any hope of letting different versions coexist.
Perhaps our convention should be to bump SO_MAJOR_VERSION for each of
our major releases and only use SO_MINOR_VERSION when we change a
library in a dot-release.

Well, the goal for coexistence is mainly the server and the
server-dependent tools (pg_dump, psql, etc.). There is no per-se
requirement to have libpq versions to coexist. If libpq has different
sonames in different releases, then they would end up in differently
named packages anyway (at least in Debian). If they don't have
different sonames, then we declare that they are compatible, so it
should be OK to have only the latest version installed. That requires
us to stay honest with the sonames, but it does not require us to
increase the sonames unnecessarily.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

According to our RELEASE_CHANGES documentation:

The major version number should be updated whenever the

source of the

library changes to make it binary incompatible. Such

changes include,

but are not limited to:

1. Removing a public function or structure (or typedef,

enum, ...)

2. Modifying a public functions arguments.

3. Removing a field from a public structure.

so while I don't think we need to update the major number for every
PostgreSQL major release, the removal of prog_name probably

required a

major bump.

Well, the point is that get_progname *isn't* a "public" function.
We never advertised it as a libpq entry point.

What this really brings out to me is that our development 
process doesn't impose a very strong boundary between libpq 
and our bundled client programs.  If the client programs were 
enforced to use only the documented public API of libpq then 
we'd not be having this discussion
--- but stuff such as libpgport support functions tends to 
slip by under the radar.  IIRC we've been bitten in exactly 
this way at least once before.  What I'm suggesting is that 
we just solve the whole class of problems permanently, by 
abandoning the assumption that we're going to guarantee 
binary compatibility across major releases.  I don't think 
that promise is really buying us anything very critical.

If we don't go that way, then we need to have some automatic
check that none of the client programs are using symbols they
shouldn't be from libpq. (Hmm ... will the existence of the
Windows port help here?)

Yes, it will. At least it will refuse to link with references that are
not in the libpqdll.def file. It won't change if the *signature* of the
functions change.

<flashback
href="http://archives.postgresql.org/pgsql-hackers-win32/2004-10/msg0004
2.php"> ;-) + thread.

//Magnus

Peter Eisentraut wrote:

Tom Lane wrote:

Well, if you just want to bump libpq's SO_MAJOR_VERSION, I won't
object.

Yes. Unless someone objects, I will do that for 8.0.* and 8.1.*.

I am thinking we should up the 8.0.* and 8.1.* releases to have the same
major number, but not make a major libpq bump for every major release
unless it is required, like with prog_name. Is that the plan?

I guess the big question is whether we would ever want a 7.2 psql or any
other client to automatically use a 7.3 libpq. In fact by upping the
major every time will 7.2 clients automatically use the 7.3 libpq or
will they have to be relinked?

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Peter Eisentraut <peter_e@gmx.net> writes:

... If they don't have
different sonames, then we declare that they are compatible, so it
should be OK to have only the latest version installed. That requires
us to stay honest with the sonames, but it does not require us to
increase the sonames unnecessarily.

Well, the problem I'm concerned about is how do we stay honest. We now
realize that our devel procedures may not catch this sort of mistake.

It might be that the Windows DLLs will catch any unintentional
dependencies, but I'm not feeling especially comfortable about it.

regards, tom lane

Tom Lane wrote:

Peter Eisentraut <peter_e@gmx.net> writes:

... If they don't have
different sonames, then we declare that they are compatible, so it
should be OK to have only the latest version installed. That requires
us to stay honest with the sonames, but it does not require us to
increase the sonames unnecessarily.

Well, the problem I'm concerned about is how do we stay honest. We now
realize that our devel procedures may not catch this sort of mistake.

It might be that the Windows DLLs will catch any unintentional
dependencies, but I'm not feeling especially comfortable about it.

I am hopefull Win32 will help here and am willing to give it another
try.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

Peter Eisentraut wrote:

Tom Lane wrote:

Well, if you just want to bump libpq's SO_MAJOR_VERSION, I won't
object.

Yes. Unless someone objects, I will do that for 8.0.* and 8.1.*.

I am thinking we should up the 8.0.* and 8.1.* releases to have the same
major number, but not make a major libpq bump for every major release
unless it is required, like with prog_name. Is that the plan?

I guess the big question is whether we would ever want a 7.2 psql or any
other client to automatically use a 7.3 libpq. In fact by upping the
major every time will 7.2 clients automatically use the 7.3 libpq or
will they have to be relinked?

Uh, if we bump up the major library version in 8.0.X, will that require
8.0.0 user applications to be recompiled? That seems worse than having
7.4.X pg apps like psql fail. They fail because they are calling
get_progname, but most user apps will not be calling that function.

Also, if we bump the major version, will an install of 8.0.X fix
problems with get_progname? Won't the 8.0.0 libpq with a higher minor
version number still prevent old psql's from running? Our install
doesn't uninstall old libraries, does it? RPM's might but I am not sure
how they do minor upgrades.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:

In fact by upping the major every time will 7.2 clients automatically use
the 7.3 libpq or will they have to be relinked?

If you do not bump the soname then 7.2 clients will automatically immediately
start using the new library when it's installed. (actually when ldconfig is
run, which distributions will normally do automatically).

If you do bump the versions then 7.2 clients will continue to use 7.2
libraries. If you have 7.2 clients, or even clients like DBD::Pg or other
third-party libpq apps then you would have to relink them to use the new
libraries. But you would probably just keep around both sets of libraries.

I could see valid arguments for either when it comes to whether to bump it
when it's not strictly necessary.

If libpq were used by lots of external applications using the public interface
then bumping it more than necessary means people have to keep around extra
versions for no reason.

On the other hand if libpq is seen as an integral part of the postgres package
and its specific behaviour is important then not bumping the version forces
admins to pick one version for everything when they might prefer that 7.2
clients stick with the 7.2 library.

In a way the fact that you release new libpq versions when you release
security releases for old releases like 7.3 or 7.4 sort of already implies an
answer. If you thought the new library was plug-in compatible and should
completely replace the old library automatically you wouldn't continue to
release modified versions of the old library, you would just release HEAD for
libpq all the time.

--
greg

Bruce Momjian wrote:

Uh, if we bump up the major library version in 8.0.X, will that
require 8.0.0 user applications to be recompiled?

No, they just keep using the old library.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

Bruce Momjian wrote:

Uh, if we bump up the major library version in 8.0.X, will that
require 8.0.0 user applications to be recompiled?

No, they just keep using the old library.

That assumes the old libraries stay around. Will they?

I am thinking the easiest solution will be to re-add get_progname() to
8.0.X and bump the major for 8.1.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073