Question and suggestion about application binary compatibility policy

Hi,

- Build an embedded SQL C application with PostgreSQL 9.2.
- Distribute the app without ecpg nor libpq libraries.  Require users
to install PostgreSQL client which includes ecpg and libpq libraries.
- Use the app with newer PostgreSQL major versions without rebuilding
the app.  That is, the users just replaces the PostgreSQL client.

I expect this is legal, because the so_major versions of ecpg and
libpq are 6 and 5 respectively for all PostgreSQL 9.x versions so
far.  However, I could not find any description of this binary
compatibility policy in the manual, so I haven't been able to answer
the customer.

Sorry I fail to understand what you mean with "legal"? Are you
wondering about license restrictions? There are none.

As for compatibility, that's what major version numbers are for. This
is not a PostgreSQL topic, but a general system one as for instance the
same holds for libc.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: pgsql-hackers-owner@postgresql.org
Sorry I fail to understand what you mean with "legal"? Are you wondering
about license restrictions? There are none.

Sorry, I just meant "correct" or "valid".

As for compatibility, that's what major version numbers are for. This is
not a PostgreSQL topic, but a general system one as for instance the same
holds for libc.

Yes, but Windows users probably don't understand or know it. So, I suggested explicitly describing the application binary compatibility policy in the PostgreSQL manual. What do you think about it?

Regards
Takayuki Tsunakawa

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Sorry I fail to understand what you mean with "legal"? Are you
wondering
about license restrictions? There are none.

Sorry, I just meant "correct" or "valid".

Ah ok, thanks for clarifying.

As for compatibility, that's what major version numbers are for.
This is
not a PostgreSQL topic, but a general system one as for instance
the same
holds for libc.

Yes, but Windows users probably don't understand or know it.  So, I
suggested explicitly describing the application binary compatibility
policy in the PostgreSQL manual.  What do you think about it?

Couldn't you point your customer to the system documentation?

Personally I don't think standard system behavior should be documented
for each application relying on it but ymmv.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Michael Meskes [mailto:meskes@postgresql.org]

Yes, but Windows users probably don't understand or know it. So, I
suggested explicitly describing the application binary compatibility
policy in the PostgreSQL manual. What do you think about it?

Couldn't you point your customer to the system documentation?

Personally I don't think standard system behavior should be documented for
each application relying on it but ymmv.

I couldn't find appropriate system documentation. Regarding Linux, I remember I saw some HOWTO on tldp.org website which explains the concept of shared library soname, but it's not very friendly for users who just want to know the application binary compatibility policy of PostgreSQL. And I don't think there's suitable documentation on Windows. Even if there is any, users will not be sure whether PostgreSQL follows those platform-specific conventions. They may have doubts about it, because even the product version "PostgreSQL x.y.z" causes misconception that x is the major version and y is the minor one.

So, I suggested documenting the compatibility policy for clarification and user friendliness as in the Oracle Database documentation below.

http://docs.oracle.com/database/121/UPGRD/app.htm#UPGRD12547

BTW, although this may be a separate topic, it may be better that we add the major version in the library names like libpq5.dll and libecpg6.dll, so that the application can fail to run with the incompatible versions of libpq and libecpg. FYI:

https://en.wikipedia.org/wiki/Side-by-side_assembly

[Excerpt]
Microsoft Visual C++ 2005 and 2008 employ SxS with all C runtime libraries. However, runtime libraries in Visual C++ 2010 no longer use this technology; instead, they include the version number of a DLL in its file name, which means that different versions of one DLL will technically be completely different DLLs now.

Any comments on these? If there's no strong objection, I think I'll submit a documentation patch in the future.

Regards
Takayuki Tsunakawa

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 31/05/2016 08:10, Tsunakawa, Takayuki wrote:

From: Michael Meskes [mailto:meskes@postgresql.org]

Yes, but Windows users probably don't understand or know it. So, I
suggested explicitly describing the application binary compatibility
policy in the PostgreSQL manual. What do you think about it?

Couldn't you point your customer to the system documentation?

Personally I don't think standard system behavior should be documented for
each application relying on it but ymmv.

I couldn't find appropriate system documentation. Regarding Linux, I remember I saw some HOWTO on tldp.org website which explains the concept of shared library soname, but it's not very friendly for users who just want to know the application binary compatibility policy of PostgreSQL. And I don't think there's suitable documentation on Windows. Even if there is any, users will not be sure whether PostgreSQL follows those platform-specific conventions. They may have doubts about it, because even the product version "PostgreSQL x.y.z" causes misconception that x is the major version and y is the minor one.

So, I suggested documenting the compatibility policy for clarification and user friendliness as in the Oracle Database documentation below.

http://docs.oracle.com/database/121/UPGRD/app.htm#UPGRD12547

BTW, although this may be a separate topic, it may be better that we add the major version in the library names like libpq5.dll and libecpg6.dll, so that the application can fail to run with the incompatible versions of libpq and libecpg. FYI:

https://en.wikipedia.org/wiki/Side-by-side_assembly

[Excerpt]
Microsoft Visual C++ 2005 and 2008 employ SxS with all C runtime libraries. However, runtime libraries in Visual C++ 2010 no longer use this technology; instead, they include the version number of a DLL in its file name, which means that different versions of one DLL will technically be completely different DLLs now.

Any comments on these? If there's no strong objection, I think I'll submit a documentation patch in the future.

Regards
Takayuki Tsunakawa

Hi,
on cygwin the postgresql binary package already include
the library versions:

/usr/bin/cygecpg-6.dll
/usr/bin/cygecpg_compat-3.dll
/usr/bin/cygpgtypes-3.dll
/usr/bin/cygpq-5.dll

attached the patch used for the build.

Regards
Marco

I couldn't find appropriate system documentation.  Regarding Linux, I
remember I saw some HOWTO on tldp.org website which explains the
concept of shared library soname, but it's not very friendly for
users who just want to know the application binary compatibility
policy of PostgreSQL.  And I don't think there's suitable

I would expect Unix sysadmins to understand that howto, but there are
others, e.g. a random hit from google: https://www.bottomupcs.com/libra
ries_and_the_linker.xhtml

There even is a wikipedia page about it: https://en.wikipedia.org/wiki/
Soname

documentation on Windows.  Even if there is any, users will not be
sure whether PostgreSQL follows those platform-specific
conventions.  They may have doubts about it, because even the product
version "PostgreSQL x.y.z" causes misconception that x is the major
version and y is the minor one.

I don't know anything about the Windows setup in PostgreSQL, but I
would find it fair to assume that PostgreSQL follows conventions.

BTW, although this may be a separate topic, it may be better that we
add the major version in the library names like libpq5.dll and
libecpg6.dll, so that the application can fail to run with the
incompatible versions of libpq and libecpg.  FYI:

Does this mean you cannot have to versions of libpq installed on the
same Windows system at the same time?

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: pgsql-hackers-owner@postgresql.org
[mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Marco Atzeri
on cygwin the postgresql binary package already include the library
versions:

/usr/bin/cygecpg-6.dll
/usr/bin/cygecpg_compat-3.dll
/usr/bin/cygpgtypes-3.dll
/usr/bin/cygpq-5.dll

attached the patch used for the build.

Thanks for the information. I didn't know there's a PostgreSQL binary package for Cygwin.

Regards
Takayuki Tsunakawa

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Michael Meskes [mailto:meskes@postgresql.org]
e.g. a random hit from google:=C2=A0https://www.bottomupcs.com/libra
ries_and_the_linker.xhtml

There even is a wikipedia page about
it:=C2=A0https://en.wikipedia.org/wiki/
Soname

Thank you for good pointers. The former is particularly nice.

BTW, although this may be a separate topic, it may be better that we
add the major version in the library names like libpq5.dll and
libecpg6.dll, so that the application can fail to run with the
incompatible versions of libpq and libecpg.=C2=A0=C2=A0FYI:

Does this mean you cannot have to versions of libpq installed on the same
Windows system at the same time?

No, you can have different versions in separate folders, as in:

C:\Program Files\PostgreSQL\9.2
C:\Program Files\PostgreSQL\9.5

and change the PATH environment variable to point to a newer version when you want to use the existing application without rebuilding it.

However, the problem I pointed out is that when the new library is incompatible with the older one, say the major version of libpq.dll changes from 5 to 6, the application user and/or developer cannot notice the incompatibility immediately and easily. On Unix/Linux, the application fails to start because the older library is not found. On the other hand, the application will start on Windows and probably cause difficult trouble due to the incompatibility.

Regards
Takayuki Tsunakawa

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 30 May 2016 at 11:04, Tsunakawa, Takayuki <
tsunakawa.takay@jp.fujitsu.com> wrote:

Hello,

I'd like to ask you about the policy of application binary compatibility.
And have a suggestion as well.

QUESTION
==================================================

My customer asked me if the following usage is correct.

- Build an embedded SQL C application with PostgreSQL 9.2.
- Distribute the app without ecpg nor libpq libraries. Require users to
install PostgreSQL client which includes ecpg and libpq libraries.

Why?

While that's probably OK, it's not especially desirable. The typical
Windows deployment model involves the application bundling all its direct
dependencies except when those are provided by a runtime redistributable
designed for that purpose.

- Use the app with newer PostgreSQL major versions without rebuilding the
app. That is, the users just replaces the PostgreSQL client.

... especially since there isn't a client-only PostgreSQL distribution
Windows.

How about adding an article about application binary compatibility in the
following section, as well as chapters for libpq, ECPG, etc?

It would be sensible to better define the binary compatibility expectations
that clients may rely upon and, when they are broken, a managed way in
which they're broken (soname bump, etc).

If you have an interest in the area it might be worth drafting a proposal
after taking a look at past binary compatibility discussions on -hackers.

There are three kinds of application assets that users are concerned about
when upgrading. Are there anything else to mention?

* libpq app
* ECPG app
* C-language user defined function (and other stuff dependent on it, such
as extensions, UDTs, etc.)

- On-disk format
- Wire protocol
- SQL-level (data types, syntax, encoding handling, settings, ...)
- ...

On the other hand, the application will start on Windows and probably

cause difficult trouble due to the incompatibility.

Yeah, we just write 'libpq.dll' on Windows.

The simplest solution would be to incorporate the soname, so it becomes
libpq0509.dll for example. Like VS does with the VS runtime. The main
downside is that because it's not a true soname and the OS has no such
concept, the linker for everything compiled against that DLL must specify
the versioned DLL name, it can't just link to 'libpq' .

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Craig Ringer
While that's probably OK, it's not especially desirable. The typical Windows deployment model involves the application bundling all its direct dependencies except when those are provided by a runtime redistributable designed for that purpose.

I think so, too, but I'm not confident that's typical. Some people may think of PostgreSQL binaries as a shared component for their applications and place it in one place, just like the PostgreSQL library package is in /usr/lib/pgsql.

- Use the app with newer PostgreSQL major versions without rebuilding the app. That is, the users just replaces the PostgreSQL client.

... especially since there isn't a client-only PostgreSQL distribution Windows.

There's a client-only installation method as follows, although I haven't checked whether EnterpriseDB, OpenSCG, or any other PostgreSQL-based products provide client-only installation.
https://www.postgresql.org/docs/devel/static/install-windows-full.html#AEN30192

[Excerpt]
--------------------------------------------------
If you want to install only the client applications and interface libraries, then you can use these commands:

install c:\destination\directory client
--------------------------------------------------

How about adding an article about application binary compatibility in the following section, as well as chapters for libpq, ECPG, etc?

It would be sensible to better define the binary compatibility expectations that clients may rely upon and, when they are broken, a managed way in which they're broken (soname bump, etc).

If you have an interest in the area it might be worth drafting a proposal after taking a look at past binary compatibility discussions on -hackers.

Sure, I'll submit a patch to pgsql-docs. Thanks to Michael's confirmation, I could answer the customer's question, so this is not an immediate task now. But I'll do.

- On-disk format
- Wire protocol
- SQL-level (data types, syntax, encoding handling, settings, ...)

Yes, I recognize these items. I omitted them because:

- On-disk format: this is handled in the PostgreSQL manual article about upgrading
- Wire protocol: I believe the compatibility will be retained
- SQL-level: ditto

But if you feel a need for their compatibility description for completeness, I'll add it. ... Yes, the explicit explanation may be necessary so that users are assured that the PostgreSQL compatibility policy matches their expectation.

The simplest solution would be to incorporate the soname, so it becomes libpq0509.dll for example. Like VS does with the VS runtime. The main downside is that because it's not a true soname and the OS has no such concept, the linker for everything compiled against that DLL must specify the versioned DLL name, it can't just link to 'libpq' .

Although I haven’t examined yet, some directive in .def file might enable applications to specify libpq.lib at build time and to link with libpq5.dll at run time.

Regards
Takayuki Tsunakawa

On 1 June 2016 at 13:09, Tsunakawa, Takayuki <tsunakawa.takay@jp.fujitsu.com

wrote:

From: pgsql-hackers-owner@postgresql.org [mailto:
pgsql-hackers-owner@postgresql.org] On Behalf Of Craig Ringer

While that's probably OK, it's not especially desirable. The typical
Windows deployment model involves the application bundling all its direct
dependencies except when those are provided by a runtime redistributable
designed for that purpose.

I think so, too, but I'm not confident that's typical. Some people may
think of PostgreSQL binaries as a shared component for their applications
and place it in one place, just like the PostgreSQL library package is in
/usr/lib/pgsql.

(Your reply formatting seems mangled, mixing my text with yours)

/usr/lib/pgsql works because *nix systems don't typically do binary
distribution.

Windows apps that rely on binary distribution should bundle the libraries
they require.

Maybe a note on windows distribution in the libpq manual would be
warranted. I thought it was so accepted as the way it's done that nobody
would really do anything else.

(Then again, EDB's installers don't bundle their Python, Perl, etc
runtimes, but I think that's partly a legal thing).

There's a client-only installation method as follows, although I haven't
checked whether EnterpriseDB, OpenSCG, or any other PostgreSQL-based
products provide client-only installation.

https://www.postgresql.org/docs/devel/static/install-windows-full.html#AEN30192

Right, and EDBs installers also let you install just the client AFAIK, but
there's no simple client library redist package, like you'd expect if it
was intended for use that way.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

However, the problem I pointed out is that when the new library is
incompatible with the older one, say the major version of libpq.dll
changes from 5 to 6, the application user and/or developer cannot
notice the incompatibility immediately and easily.  On Unix/Linux,
the application fails to start because the older library is not
found.  On the other hand, the application will start on Windows and
probably cause difficult trouble due to the incompatibility.

I don't think this is a very good situation, but I have no idea if this
can be solved. However, I'd prefer a technical solution over a
documentation one.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Meskes at (Debian|Postgresql) dot Org
Jabber: michael at xmpp dot meskes dot org
VfL Borussia! Força Barça! SF 49ers! Use Debian GNU/Linux, PostgreSQL

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Mon, May 30, 2016 at 03:04:24AM +0000, Tsunakawa, Takayuki wrote:

Hello,

I'd like to ask you about the policy of application binary compatibility. And have a suggestion as well.

QUESTION
==================================================

My customer asked me if the following usage is correct.

- Build an embedded SQL C application with PostgreSQL 9.2.
- Distribute the app without ecpg nor libpq libraries. Require users to install PostgreSQL client which includes ecpg and libpq libraries.
- Use the app with newer PostgreSQL major versions without rebuilding the app. That is, the users just replaces the PostgreSQL client.

I expect this is legal, because the so_major versions of ecpg and libpq are 6 and 5 respectively for all PostgreSQL 9.x versions so far. However, I could not find any description of this binary compatibility policy in the manual, so I haven't been able to answer the customer.

What's the official policy of application binary compatibility? I found the same discussion in the below thread, but I'm afraid any clear answer wasn't given:

/messages/by-id/522F0B6B.1040006@4js.com

SUGGESTION
==================================================

How about adding an article about application binary compatibility in the following section, as well as chapters for libpq, ECPG, etc?

17.6. Upgrading a PostgreSQL Clust
https://www.postgresql.org/docs/devel/static/upgrading.html

There are three kinds of application assets that users are concerned about when upgrading. Are there anything else to mention?

* libpq app
* ECPG app
* C-language user defined function (and other stuff dependent on it, such as extensions, UDTs, etc.)

I know this thread is old but it bounced around a lot of ideas. I think
there are some open questions:

* Will a new application link against an older minor-version libpq?
* Will an old application link against a newer minor-version libpq?

I think we are all in agreement that a binary cannot run using a libpq
with a different major version number.

We have this text in src/tools/RELEASE_CHANGES:

Major Version
=============

The major version number should be updated whenever the source of the
library changes to make it binary incompatible. Such changes include,
but are not limited to:

1. Removing a public function or structure (or typedef, enum, ...)

2. Modifying a public functions arguments.

3. Removing a field from a public structure.

4. Adding a field to a public structure, unless steps have been
previously taken to shield users from such a change, for example by
such structures only ever being allocated/instantiated by a library
function which would give the new field a suitable default value.

Adding a new function should NOT force an increase in the major version
number. (Packagers will see the standard minor number update and install
the new library.) When the major version is increased all applications
which link to the library MUST be recompiled - this is not desirable. When
the major version is updated the minor version gets reset.

Minor Version
=============

The minor version number should be updated whenever the functionality of
the library has changed, typically a change in source code between releases
would mean an increase in the minor version number so long as it does not
require a major version increase.

Given that we make at least minor changes to our libraries in every major
PostgreSQL version, we always bump all minor library version numbers at the
start of each development cycle as a matter of policy.

This is saying running against a mismatched minor version should be fine
for a binary.

ecpg is a little tricker because it has knowledge of SQL data types and
might not support certain features if the ecpg library isn't _run_
against the same major server version. My guess is that older ecpg
libraries will run fine against newer servers, but the opposite might
not be true.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

From: Bruce Momjian [mailto:bruce@momjian.us]
We have this text in src/tools/RELEASE_CHANGES:
...
This is saying running against a mismatched minor version should be fine
for a binary.

Thanks for a good rationale.

I know this thread is old but it bounced around a lot of ideas. I think
there are some open questions:

* Will a new application link against an older minor-version libpq?
* Will an old application link against a newer minor-version libpq?

The former does not always hold true, if the application uses a new libpq function which is not in an old miner-version. But I think the backward-compatibility is enough.

Regards
Takayuki Tsunakawa

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Thu, Jun 23, 2016 at 06:42:57AM +0000, Tsunakawa, Takayuki wrote:

From: Bruce Momjian [mailto:bruce@momjian.us]
We have this text in src/tools/RELEASE_CHANGES:
...
This is saying running against a mismatched minor version should be fine
for a binary.

Thanks for a good rationale.

I know this thread is old but it bounced around a lot of ideas. I think
there are some open questions:

* Will a new application link against an older minor-version libpq?
* Will an old application link against a newer minor-version libpq?

The former does not always hold true, if the application uses a new libpq function which is not in an old miner-version. But I think the backward-compatibility is enough.

Yes, I think that is correct, and I think that is covered in the file
posted:

Adding a new function should NOT force an increase in the major version
number.

--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers