Proposal: Document ABI Compatibility

Started by David E. Wheelerover 1 year ago46 messages
#1David E. Wheeler
david@justatheory.com

Hackers,

At the PGConf Unconference session on improving extension support in core, we talked quite a bit about the recent anxiety among extension developers about a lack of an ABI compatibility guarantee in Postgres. Yurii Rashkovskii did a little header file spelunking and talked[1]https://www.pgevents.ca/events/pgconfdev2024/schedule/session/14 about a few changes in minor version releases, including to apparent field order in structs. Jeremy Schneider posted the example on Twitter[2]https://x.com/jer_s/status/1785717368804815026, and Peter G replied[3]https://x.com/petervgeoghegan/status/1785720228237717627:

You must be referring to my commit 714780dc. The new field is stored within alignment padding (though only on back branches). Has this been tied to a known problem?

At the Unconference, Tom Lane said that this approach is pretty well drilled into the heads of every committer, and new ones pick it up through experience. The goal, IIUC, is to never introduce binary incompatibilities into the C APIs in minor releases. This standard would be good to document, to let extension developers know exactly what the guarantees are.

I’m happy to start a doc patch to add an ABI compatibility guarantee (presumably in xfunc.sgml), but want to be sure the details are right, namely:

* There are no source or binary compatibility guarantees for major releases.

* The ABI is guaranteed to change only in backward compatible ways in minor releases. If for some reason it doesn’t it’s a bug that will need to be fixed.

This ensures that an extension compiled against an earlier minor release will continue to work without recompilation on later minor releases of the same major version.

But if I understand correctly, the use of techniques like adding a new field in padding does not mean that extensions compiled on later minor releases will work on earlier minor releases of the same major version. Unless, that is, we can provide a complete list of things not to do (like make use of padding) to avoid it. Is that feasible?

Are there other details it should include?

Thanks,

David

[1]: https://www.pgevents.ca/events/pgconfdev2024/schedule/session/14
[2]: https://x.com/jer_s/status/1785717368804815026
[3]: https://x.com/petervgeoghegan/status/1785720228237717627

#2Andres Freund
andres@anarazel.de
In reply to: David E. Wheeler (#1)
Re: Proposal: Document ABI Compatibility

Hi,

On 2024-06-03 14:43:17 -0400, David E. Wheeler wrote:

At the PGConf Unconference session on improving extension support in core,
we talked quite a bit about the recent anxiety among extension developers
about a lack of an ABI compatibility guarantee in Postgres.

Are there notes for the session?

Yurii Rashkovskii did a little header file spelunking and talked[1] about a
few changes in minor version releases, including to apparent field order in
structs.

It'd be nice if the slides for the talk could be uploaded...

You must be referring to my commit 714780dc. The new field is stored within alignment padding (though only on back branches). Has this been tied to a known problem?

At the Unconference, Tom Lane said that this approach is pretty well drilled
into the heads of every committer, and new ones pick it up through
experience. The goal, IIUC, is to never introduce binary incompatibilities
into the C APIs in minor releases. This standard would be good to document,
to let extension developers know exactly what the guarantees are.

I don't think we can really make this a hard guarantee. Yes, we try hard to
avoid ABI breaks, but there IIRC have been a few cases over the years where
that wasn't practical for some reason. If we have to decide between a bad bug
and causing an ABI issue that's unlikely to affect anybody, we'll probably
choose the ABI issue.

* The ABI is guaranteed to change only in backward compatible ways in minor
releases. If for some reason it doesn’t it’s a bug that will need to be
fixed.

Thus I am not really on board with this statement as-is.

Extensions in general can do lots of stuff, guaranteeing that bug fixes don't
cause any problems is just not feasible.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

But if I understand correctly, the use of techniques like adding a new field
in padding does not mean that extensions compiled on later minor releases
will work on earlier minor releases of the same major version.

I don't think it's common for such new-fields-in-padding to cause problems
when using an earlier minor PG version. For that the extension would need to
actually rely on the presence of the new field, but typically that'd not be
the case when we introduce a new field in a minor version.

Unless, that is, we can provide a complete list of things not to do (like
make use of padding) to avoid it. Is that feasible?

You can't really rely on the contents of padding, in general. So I don't think
this is really something that needs to be called out.

Greetings,

Andres Freund

#3David E. Wheeler
david@justatheory.com
In reply to: Andres Freund (#2)
Re: Proposal: Document ABI Compatibility

On Jun 3, 2024, at 14:58, Andres Freund <andres@anarazel.de> wrote:

Hi,

Hello Andres.

Are there notes for the session?

Yes, but not posted yet. Here’s what Andreas 'ads' Scherbaum sent me for that bit of the conversation:

* Core is focused on core ABI stability
* David: No "statement of stability" in Core
* David/Jeremy/Tom: coding guidelines, style guidelines
* useful to have docs in core about what's stable and what's not, what you should compile against or not, and ABI guarantees
* Abigale: there are hooks, but no overall concept for extensions
* Tom: Peter Eisentraut is working on tests for extensions stability
* Jeremy: nothing is preventing people from installing incompatible versions

It'd be nice if the slides for the talk could be uploaded...

He also talked about it at Mini-Summit Five[1]https://justatheory.com/2024/05/mini-summit-five/, for which I posted slides[2]https://justatheory.com/shared/extension-ecosystem-summit/omni-universally-buildable-extensions.pdf (slides 11-14) and video[3]https://youtu.be/R5ijx8IJyaM (starting at 29:08).

I don't think we can really make this a hard guarantee. Yes, we try hard to
avoid ABI breaks, but there IIRC have been a few cases over the years where
that wasn't practical for some reason. If we have to decide between a bad bug
and causing an ABI issue that's unlikely to affect anybody, we'll probably
choose the ABI issue.

We can document that too, and perhaps a policy for letting people know. I thought I recalled something like this in the past, but Rob Treat did some spelunking through change logs and found only CVEs that needed repairs by manually running some SQL. So some sense of its rarity would also be useful.

Thus I am not really on board with this statement as-is.

That’s fine, we should get it to where there’s consensus on the ordering and agreement on what, exactly, it should be.

Extensions in general can do lots of stuff, guaranteeing that bug fixes don't
cause any problems is just not feasible.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

In the community Slack[4]https://pgtreats.info/slack-invite, Matthias van de Meent writes[5]https://postgresteam.slack.com/archives/C056ZA93H1A/p1716502630690559?thread_ts=1716500801.036709&amp;cid=C056ZA93H1A:

Citus’ pg_version_compat.h[7] where it re-implements a low-level function that was newly introduced in PG14.7. If you build against PG14.7+ headers, you may get random crashes when running on 14.6.

I suppose it would work fine on 14.7 if compiled on 14.6 though. I suspect there aren’t many examples, though, mostly just a lot of anxiety, and some have decided that extensions must be recompiled for every minor release in order to avoid the issue. StackGres[7]https://stackgres.io/doc/latest/intro/extensions/ is one example, but I suspect Omni (Yurii’s company) may follow.

I don't think it's common for such new-fields-in-padding to cause problems
when using an earlier minor PG version. For that the extension would need to
actually rely on the presence of the new field, but typically that'd not be
the case when we introduce a new field in a minor version.

That’s what I was thinking, so “compatibility assured only if you don’t use padding yourself” would be super helpful.

Unless, that is, we can provide a complete list of things not to do (like
make use of padding) to avoid it. Is that feasible?

You can't really rely on the contents of padding, in general. So I don't think
this is really something that needs to be called out.

Sure, probably not a problem, but if that’s the sole qualifier for making binary changes, I think it’s worth saying, as opposed to “we don’t make any”. Something like “Only changes to padding, which you never used anyway, right?” :-)

D

[1]: https://justatheory.com/2024/05/mini-summit-five/
[2]: https://justatheory.com/shared/extension-ecosystem-summit/omni-universally-buildable-extensions.pdf
[3]: https://youtu.be/R5ijx8IJyaM
[4]: https://pgtreats.info/slack-invite
[5]: https://postgresteam.slack.com/archives/C056ZA93H1A/p1716502630690559?thread_ts=1716500801.036709&amp;cid=C056ZA93H1A
[6]: https://github.com/citusdata/citus/blob/main/src/include/pg_version_compat.h#L236-L248
[7]: https://stackgres.io/doc/latest/intro/extensions/

#4Tom Lane
tgl@sss.pgh.pa.us
In reply to: Andres Freund (#2)
Re: Proposal: Document ABI Compatibility

Andres Freund <andres@anarazel.de> writes:

On 2024-06-03 14:43:17 -0400, David E. Wheeler wrote:

* The ABI is guaranteed to change only in backward compatible ways in minor
releases. If for some reason it doesn’t it’s a bug that will need to be
fixed.

Thus I am not really on board with this statement as-is.

Me either. There are degrees of ABI compatibility, and we'll choose
the least invasive way, but it's seldom the case that no conceivable
extension will be broken. For example, if we can't squeeze a new
field into padding space, we'll typically put it at the end of the
struct in existing branches. That's okay unless some extension has
a dependency on sizeof(the struct), for instance because it's
allocating such structs itself. Also, for either the padding-space
or add-at-the-end approaches, we're probably in trouble if some
extension is creating its own instances of such structs, because
it will not know how to fill the new field properly. We try not to
change structs that we think extensions are likely to create ...
but that's a guess not a guarantee.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

Yes, that could be a fruitful discussion.

You can't really rely on the contents of padding, in general. So I don't think
this is really something that needs to be called out.

For node structs, padding will generally be zero because we memset
them to zeroes. So to the extent that zero is an okay value for
such a new field, that can help --- but if zero were always okay
then we'd likely not need a new field in the first place.

regards, tom lane

#5David Christensen
david+pg@pgguru.net
In reply to: David E. Wheeler (#3)
Re: Proposal: Document ABI Compatibility

I don't think it's common for such new-fields-in-padding to cause problems
when using an earlier minor PG version. For that the extension would need to
actually rely on the presence of the new field, but typically that'd not be
the case when we introduce a new field in a minor version.

That’s what I was thinking, so “compatibility assured only if you don’t use padding yourself” would be super helpful.

I was under the impression that the (a?) concern was related to
compiling the newer version and using that against an older version,
so if you always compiled the extension against the latest point
release, it wouldn't necessarily be backwards-compatible with older
installations. (While probably allocated padding is zero'd out in the
old version, I'd not guarantee that that's the case.)

Unless, that is, we can provide a complete list of things not to do (like
make use of padding) to avoid it. Is that feasible?

You can't really rely on the contents of padding, in general. So I don't think
this is really something that needs to be called out.

Sure, probably not a problem, but if that’s the sole qualifier for making binary changes, I think it’s worth saying, as opposed to “we don’t make any”. Something like “Only changes to padding, which you never used anyway, right?” :-)

Wonder if what might be more useful is some sort of distribution list
for extensions authors to be notified of specific compatibility
changes. Could be powered by parsing commit messages for conventions
about backwards-incompatible changes; padding usage could be one,
maybe there are others we could code for. (For padding, imagine a
tool that is able to look at struct offsets between git revisions and
note any offset differences/changes here that extensions authors could
be aware of; even diffs of `pahole` output between revisions could be
helpful.)

ABI guarantees for extensions are hard because all of core *is*
potentially the ABI (or at least if you are well-behaved, public
functions and structs), so some sort of "these interfaces won't break
in minor releases but use other internals and you're on your own"
might be useful distinction. (We discussed trying to classify APIs
with varying levels of support, but that also comes with its own set
of issues/overhead/maintenance for code authors/committers.)

#6Andres Freund
andres@anarazel.de
In reply to: David E. Wheeler (#3)
Re: Proposal: Document ABI Compatibility

Hi,

On 2024-06-03 15:21:04 -0400, David E. Wheeler wrote:

Extensions in general can do lots of stuff, guaranteeing that bug fixes don't
cause any problems is just not feasible.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

In the community Slack[4], Matthias van de Meent writes[5]:

Citus’ pg_version_compat.h[7] where it re-implements a low-level function that was newly introduced in PG14.7. If you build against PG14.7+ headers, you may get random crashes when running on 14.6.

I don't see how this would trigger random crashes.

Unfortunately [4] doesn't seem to take me to a relevant message (pruned chat
history?), so I can't infer more from that context.

I suppose it would work fine on 14.7 if compiled on 14.6 though. I suspect
there aren’t many examples, though, mostly just a lot of anxiety, and some
have decided that extensions must be recompiled for every minor release in
order to avoid the issue. StackGres[7] is one example, but I suspect Omni
(Yurii’s company) may follow.

Regardless of ABI issues, it's probably a good idea to continually run tests
against in-development minor versions, just to prevent something breaking from
creeping in. IIRC there were a handful of cases where we accidentally broke
some extension, because they relied on some implementation details.

Unless, that is, we can provide a complete list of things not to do (like
make use of padding) to avoid it. Is that feasible?

You can't really rely on the contents of padding, in general. So I don't think
this is really something that needs to be called out.

Sure, probably not a problem, but if that’s the sole qualifier for making
binary changes, I think it’s worth saying, as opposed to “we don’t make
any”. Something like “Only changes to padding, which you never used anyway,
right?” :-)

IDK, to me something like this seems to promise more than we actually can.

Greetings,

Andres Freund

#7David E. Wheeler
david@justatheory.com
In reply to: Andres Freund (#6)
Re: Proposal: Document ABI Compatibility

On Jun 3, 2024, at 5:56 PM, Andres Freund <andres@anarazel.de> wrote:

I don't see how this would trigger random crashes.

Unfortunately [4] doesn't seem to take me to a relevant message (pruned chat
history?), so I can't infer more from that context.

You can use [4]https://pgtreats.info/slack-invite to join the Slack (if you haven’t already) and [5]https://postgresteam.slack.com/archives/C056ZA93H1A/p1716502630690559?thread_ts=1716500801.036709&amp;cid=C056ZA93H1A for the relevant post.

Regardless of ABI issues, it's probably a good idea to continually run tests
against in-development minor versions, just to prevent something breaking from
creeping in. IIRC there were a handful of cases where we accidentally broke
some extension, because they relied on some implementation details.

Oh yeah, I run regular tests against the latest minor release of all supported Postgres version for my extensions, using pgxn-tools[6]https://github.com/pgxn/docker-pgxn-tools, which looks like this[7]https://github.com/pgxn/docker-pgxn-tools/actions/runs/9351752462. Which I consider absolutely essential. But it doesn’t mean that something compiled against .4 will work with .3 and vice versa. That’s what we could use the guidance/guarantees on.

Sure, probably not a problem, but if that’s the sole qualifier for making
binary changes, I think it’s worth saying, as opposed to “we don’t make
any”. Something like “Only changes to padding, which you never used anyway,
right?” :-)

IDK, to me something like this seems to promise more than we actually can.

What I’d like to do is figure out exactly what we *can* promise and perhaps some guidelines, and start with that.

Best,

David

[4]: https://pgtreats.info/slack-invite
[5]: https://postgresteam.slack.com/archives/C056ZA93H1A/p1716502630690559?thread_ts=1716500801.036709&amp;cid=C056ZA93H1A
[6]: https://github.com/pgxn/docker-pgxn-tools
[7]: https://github.com/pgxn/docker-pgxn-tools/actions/runs/9351752462

#8Laurenz Albe
laurenz.albe@cybertec.at
In reply to: Tom Lane (#4)
Re: Proposal: Document ABI Compatibility

On Mon, 2024-06-03 at 15:38 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2024-06-03 14:43:17 -0400, David E. Wheeler wrote:

* The ABI is guaranteed to change only in backward compatible ways in minor
releases. If for some reason it doesn’t it’s a bug that will need to be
fixed.

Thus I am not really on board with this statement as-is.

Me either.  There are degrees of ABI compatibility, and we'll choose
the least invasive way, but it's seldom the case that no conceivable
extension will be broken.

oracle_fdw has been broken by minor releases several times in the past.
This may well be because of weird things that I am doing; still, my
experience is that minor releases are not always binary compatible.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

Yes, that could be a fruitful discussion.

Digging through my commits brought up 6214e2b2280462cbc3aa1986e350e167651b3905,
for one.

Yours,
Laurenz Albe

#9Peter Eisentraut
peter@eisentraut.org
In reply to: Laurenz Albe (#8)
Re: Proposal: Document ABI Compatibility

On 04.06.24 02:11, Laurenz Albe wrote:

On Mon, 2024-06-03 at 15:38 -0400, Tom Lane wrote:

Andres Freund <andres@anarazel.de> writes:

On 2024-06-03 14:43:17 -0400, David E. Wheeler wrote:

* The ABI is guaranteed to change only in backward compatible ways in minor
releases. If for some reason it doesn’t it’s a bug that will need to be
fixed.

Thus I am not really on board with this statement as-is.

Me either.  There are degrees of ABI compatibility, and we'll choose
the least invasive way, but it's seldom the case that no conceivable
extension will be broken.

oracle_fdw has been broken by minor releases several times in the past.
This may well be because of weird things that I am doing; still, my
experience is that minor releases are not always binary compatible.

It'd be interesting to see a few examples of actual minor-version-upgrade
extension breakages, so we can judge what caused them.

Yes, that could be a fruitful discussion.

Digging through my commits brought up 6214e2b2280462cbc3aa1986e350e167651b3905,
for one.

I'm not sure I can see how that would have broken oracle_fdw, but in any
case it's an interesting example. This patch did not change any structs
incompatibly, but it changed the semantics of a function without
changing the name:

  extern void InitResultRelInfo(ResultRelInfo *resultRelInfo,
                               Relation resultRelationDesc,
                               Index resultRelationIndex,
-                             Relation partition_root,
+                             ResultRelInfo *partition_root_rri,
                               int instrument_options);

If an extension calls this function, something would possibly crash if
it's on the wrong side of the update fence.

This could possibly be avoided by renaming the symbol in backbranches.
Maybe something like

#define InitResultRelInfo InitResultRelInfo2

Then you'd get a specific error message when loading the module, rather
than a crash.

This might be something to consider:

no ABI break is better than an explicit ABI break is better than a
silent ABI break

(Although this is actually an API break, isn't it?)

#10David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#9)
Re: Proposal: Document ABI Compatibility

On Jun 4, 2024, at 03:18, Peter Eisentraut <peter@eisentraut.org> wrote:

This could possibly be avoided by renaming the symbol in backbranches. Maybe something like

#define InitResultRelInfo InitResultRelInfo2

Then you'd get a specific error message when loading the module, rather than a crash.

That sounds more useful, yes. Is that a practice the project would consider adopting?

There’s also oracle_fdw@d137d15[1]https://github.com/laurenz/oracle_fdw/commit/d137d15edca8c67df1e5cccca01f417f4833b028, which says:

An API break in PostgreSQL 10.4 and 9.6.9 makes it impossible
to use these versions: the "extract_actual_join_clauses" function
gained an additional parameter.

The 10.4 commit is 68fab04, and it does indeed add a new function:

``` patch
--- a/src/include/optimizer/restrictinfo.h
+++ b/src/include/optimizer/restrictinfo.h
@@ -36,6 +36,7 @@ extern List *get_actual_clauses(List *restrictinfo_list);
 extern List *extract_actual_clauses(List *restrictinfo_list,
                       bool pseudoconstant);
 extern void extract_actual_join_clauses(List *restrictinfo_list,
+                           Relids joinrelids,
                            List **joinquals,
                            List **otherquals);
 extern bool join_clause_is_movable_to(RestrictInfo *rinfo, RelOptInfo *baserel);
```

I wonder if that sort of change could be avoided in backpatches, maybe by adding and using a `extract_actual_join_clauses_compat` function and using that internally instead?

Or, to David C’s point, perhaps it would be better to say there are some categories of APIs that are not subject to any guarantees in minor releases?

Best,

David

[1]: https://github.com/laurenz/oracle_fdw/commit/d137d15edca8c67df1e5cccca01f417f4833b028
[2]: https://github.com/postgres/postgres/commit/68fab04f7c2a07c5308e3d2957198ccd7a80ebc5#diff-bb6fa74cb115e19684092f0938131cd5d99b26fa2d49480f7ea7f28e937a7fb4

#11Andres Freund
andres@anarazel.de
In reply to: David E. Wheeler (#10)
Re: Proposal: Document ABI Compatibility

Hi,

On 2024-06-10 15:05:32 -0400, David E. Wheeler wrote:

An API break in PostgreSQL 10.4 and 9.6.9 makes it impossible
to use these versions: the "extract_actual_join_clauses" function
gained an additional parameter.

The 10.4 commit is 68fab04, and it does indeed add a new function:

That's 6 years ago, not sure we can really learn that much from that.

And it's not like it's actually impossible, #ifdefs aren't great, but they are
better than nothing.

``` patch
--- a/src/include/optimizer/restrictinfo.h
+++ b/src/include/optimizer/restrictinfo.h
@@ -36,6 +36,7 @@ extern List *get_actual_clauses(List *restrictinfo_list);
extern List *extract_actual_clauses(List *restrictinfo_list,
bool pseudoconstant);
extern void extract_actual_join_clauses(List *restrictinfo_list,
+                           Relids joinrelids,
List **joinquals,
List **otherquals);
extern bool join_clause_is_movable_to(RestrictInfo *rinfo, RelOptInfo *baserel);
```

I wonder if that sort of change could be avoided in backpatches, maybe by adding and using a `extract_actual_join_clauses_compat` function and using that internally instead?

Or, to David C’s point, perhaps it would be better to say there are some categories of APIs that are not subject to any guarantees in minor releases?

I'm honestly very dubious that this is a good point to introduce a bunch of
formalism. It's a already a lot of work to maintain them, if we make it even
harder we'll end up more fixes not being backported, because it's not worth
the pain.

To be blunt, the number of examples raised here doesn't seem to indicate that
this is an area where we need to invest additional resources. We are already
severely constrained as a project by committer bandwidth, there are plenty
other things that seem more important to focus on.

Greetings,

Andres Freund

#12David E. Wheeler
david@justatheory.com
In reply to: Andres Freund (#11)
Re: Proposal: Document ABI Compatibility

On Jun 10, 2024, at 15:39, Andres Freund <andres@anarazel.de> wrote:

That's 6 years ago, not sure we can really learn that much from that.

And it's not like it's actually impossible, #ifdefs aren't great, but they are
better than nothing.

Right, it’s just that extension authors could use some notification that such a change is coming so they can update their code, if necessary.

Or, to David C’s point, perhaps it would be better to say there are some categories of APIs that are not subject to any guarantees in minor releases?

I'm honestly very dubious that this is a good point to introduce a bunch of
formalism. It's a already a lot of work to maintain them, if we make it even
harder we'll end up more fixes not being backported, because it's not worth
the pain.

Well it’s a matter of distributing the work. I don’t want to increase anyone’s workload unnecessarily, but as it is stuff like this can be surprising to extension maintainers with some expectation of minor release stability who had no warning of the change. That kind of thing can dissuade some people from deciding to write or maintain extensions, and lead others to recompile and distribute binaries for every single minor release.

To be blunt, the number of examples raised here doesn't seem to indicate that
this is an area where we need to invest additional resources. We are already
severely constrained as a project by committer bandwidth, there are plenty
other things that seem more important to focus on.

So my question is, what’s the least onerous thing for committers to commit to doing that we can write down to properly set expectations? That’s where I want to start: can we publish a policy that reflects what committers already adhere to? And is there some way to let people know that an incompatible change is being released? Even if it just starts out in the release notes?

Based on this thread, I’ve drafted the sort of policy I have in mind. Please don’t assume I’m advocating for exactly the wording here! Let’s workshop this until it’s something the committers and core team can agree to. (At that point I’ll turn it into a doc patch) Have a look and let me know what you think.

``` md

ABI Policy
==========

The PostgreSQL core team maintains two application binary interface (ABI) guarantees: one for major releases and one for minor releases.

Major Releases
--------------

Applications that use the PostgreSQL APIs must be compiled for each major release supported by the application. The inclusion of `PG_MODULE_MAGIC` ensures that code compiled for one major version will rejected by other major versions.

Furthermore, new releases may make API changes that require code changes. Use the `PG_VERSION_NUM` constant to adjust code in a backwards compatible way:

``` c
#if PG_VERSION_NUM >= 160000
#include "varatt.h"
#endif
```

PostgreSQL avoids unnecessary API changes in major releases, but usually ships a few necessary API changes, including deprecation, renaming, and argument variation. In such cases the incompatible changes will be listed in the Release Notes.

Minor Releases
--------------

PostgreSQL makes every effort to avoid both API and ABI breaks in minor releases. In general, an application compiled against any minor release will work with any other minor release, past or future.

When a change *is* required, PostgreSQL will choose the least invasive way possible, for example by squeezing a new field into padding space or appending it to the end of a struct. This sort of change should not impact dependent applications unless they use `sizeof(the struct)` or create their own instances of such structs --- patterns best avoided.

In rare cases, however, even such non-invasive changes may be impractical or impossible. In such an event, the change will be documented in the Release Notes, and details on the issue will also be posted to [TBD; mail list? Blog post? News item?].

The project strongly recommends that developers adopt continuous integration testing at least for the latest minor release all major versions of Postgres they support.
```

Best,

David

#13Peter Eisentraut
peter@eisentraut.org
In reply to: David E. Wheeler (#12)
Re: Proposal: Document ABI Compatibility

On 11.06.24 16:55, David E. Wheeler wrote:

On Jun 10, 2024, at 15:39, Andres Freund <andres@anarazel.de> wrote:

That's 6 years ago, not sure we can really learn that much from that.

And it's not like it's actually impossible, #ifdefs aren't great, but they are
better than nothing.

Right, it’s just that extension authors could use some notification that such a change is coming so they can update their code, if necessary.

I think since around 6 years ago we have been much more vigilant about
avoiding ABI breaks. So if there aren't any more recent examples of
breakage, then maybe that was ultimately successful, and the upshot is,
continue to be vigilant at about the same level?

#14Jelte Fennema-Nio
postgres@jeltef.nl
In reply to: Peter Eisentraut (#13)
Re: Proposal: Document ABI Compatibility

On Wed, 12 Jun 2024 at 14:44, Peter Eisentraut <peter@eisentraut.org> wrote:

I think since around 6 years ago we have been much more vigilant about
avoiding ABI breaks. So if there aren't any more recent examples of
breakage, then maybe that was ultimately successful, and the upshot is,
continue to be vigilant at about the same level?

While not strictly an ABI break I guess, the backport of 32d5a4974c81
broke building Citus against 13.10 and 14.7[1]https://github.com/citusdata/citus/pull/6711.

[1]: https://github.com/citusdata/citus/pull/6711

#15David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#13)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 8:43 AM, Peter Eisentraut <peter@eisentraut.org> wrote:

Right, it’s just that extension authors could use some notification that such a change is coming so they can update their code, if necessary.

I think since around 6 years ago we have been much more vigilant about avoiding ABI breaks. So if there aren't any more recent examples of breakage, then maybe that was ultimately successful, and the upshot is, continue to be vigilant at about the same level?

That sounds great to me. I’d like to get it documented, though, so that extension and other third party developers are aware of it, and not just making wild guesses and scaring each other over (perhaps) misconceptions.

D

#16David E. Wheeler
david@justatheory.com
In reply to: Jelte Fennema-Nio (#14)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 8:58 AM, Jelte Fennema-Nio <postgres@jeltef.nl> wrote:

While not strictly an ABI break I guess, the backport of 32d5a4974c81
broke building Citus against 13.10 and 14.7[1].

[1]: https://github.com/citusdata/citus/pull/6711

Interesting one. We might want to advise projects to use deferent names if they copy code from the core, use an extension-specific prefix perhaps. That way if it gets backported by the core, as in this example, it won’t break anything, and the extension can choose to switch.

D

#17Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#4)
Re: Proposal: Document ABI Compatibility

On Mon, Jun 3, 2024 at 3:39 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Me either. There are degrees of ABI compatibility

Exactly this!

What I think would be useful to document is our usual practices e.g.
adding new struct members at the end of structs, trying to avoid
changing public function signatures. If we document promises to
extension authors, I don't know how much difference that will make:
we'll probably end up needing to violate them at some point for one
reason or another. But if we document what committers should do, then
we might do better than we're now, because committers will be more
likely to do it right, and extension authors can also read those
instructions to understand what our practices are.

--
Robert Haas
EDB: http://www.enterprisedb.com

#18David E. Wheeler
david@justatheory.com
In reply to: Robert Haas (#17)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 10:47, Robert Haas <robertmhaas@gmail.com> wrote:

What I think would be useful to document is our usual practices e.g.
adding new struct members at the end of structs, trying to avoid
changing public function signatures. If we document promises to
extension authors, I don't know how much difference that will make:
we'll probably end up needing to violate them at some point for one
reason or another.

I think that’s fine if there is some sort of notification process. The policy I drafted upthread starts with making sure the such a break is mentioned in the release notes.

But if we document what committers should do, then
we might do better than we're now, because committers will be more
likely to do it right, and extension authors can also read those
instructions to understand what our practices are.

Yes, this, thank you!

D

#19Andres Freund
andres@anarazel.de
In reply to: David E. Wheeler (#12)
Re: Proposal: Document ABI Compatibility

On 2024-06-11 10:55:38 -0400, David E. Wheeler wrote:

ABI Policy
==========

The PostgreSQL core team maintains two application binary interface (ABI) guarantees: one for major releases and one for minor releases.

I.e. for major versions it's "there is none"?

Major Releases
--------------

Applications that use the PostgreSQL APIs must be compiled for each major release supported by the application. The inclusion of `PG_MODULE_MAGIC` ensures that code compiled for one major version will rejected by other major versions.

Furthermore, new releases may make API changes that require code changes. Use the `PG_VERSION_NUM` constant to adjust code in a backwards compatible way:

``` c
#if PG_VERSION_NUM >= 160000
#include "varatt.h"
#endif
```

PostgreSQL avoids unnecessary API changes in major releases, but usually
ships a few necessary API changes, including deprecation, renaming, and
argument variation.

In such cases the incompatible changes will be listed in the Release Notes.

I don't think we actually exhaustively list all of them.

Minor Releases
--------------

PostgreSQL makes every effort to avoid both API and ABI breaks in minor releases. In general, an application compiled against any minor release will work with any other minor release, past or future.

s/every/a reasonable/ or just s/every/an/

When a change *is* required, PostgreSQL will choose the least invasive way
possible, for example by squeezing a new field into padding space or
appending it to the end of a struct. This sort of change should not impact
dependent applications unless they use `sizeof(the struct)` or create their
own instances of such structs --- patterns best avoided.

The padding case doesn't affect sizeof() fwiw.

I think there's too often not an alternative to using sizeof(), potentially
indirectly (via makeNode() or such. So this sounds a bit too general.

Greetings,

Andres Freund

#20Andres Freund
andres@anarazel.de
In reply to: Jelte Fennema-Nio (#14)
Re: Proposal: Document ABI Compatibility

Hi,

On 2024-06-12 14:58:04 +0200, Jelte Fennema-Nio wrote:

On Wed, 12 Jun 2024 at 14:44, Peter Eisentraut <peter@eisentraut.org> wrote:

I think since around 6 years ago we have been much more vigilant about
avoiding ABI breaks. So if there aren't any more recent examples of
breakage, then maybe that was ultimately successful, and the upshot is,
continue to be vigilant at about the same level?

While not strictly an ABI break I guess, the backport of 32d5a4974c81
broke building Citus against 13.10 and 14.7[1].

I think that kind of thing is not something we (PG devs) really can do
anything about. It's also
a) fairly easy thing to fix
b) fails during compilation
c) doesn't break ABI afaict

Greetings,

Andres Freund

In reply to: Tom Lane (#4)
Re: Proposal: Document ABI Compatibility

On Mon, Jun 3, 2024 at 3:38 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Thus I am not really on board with this statement as-is.

Me either. There are degrees of ABI compatibility, and we'll choose
the least invasive way, but it's seldom the case that no conceivable
extension will be broken. For example, if we can't squeeze a new
field into padding space, we'll typically put it at the end of the
struct in existing branches. That's okay unless some extension has
a dependency on sizeof(the struct), for instance because it's
allocating such structs itself.

Right. While there certainly are code bases (mostly C libraries
without a huge surface area) where taking the hardest possible line on
ABI breakage makes sense, and where ABI breakage can be detected by a
mechanical process, that isn't us. In fact I'd say that Postgres is
just about the further possible thing from that.

I'm a little surprised that we don't seem to have all that many
problems with ABI breakage, though. Although we theoretically have a
huge number of APIs that extension authors might choose to use, that
isn't really true in practical terms. The universe of theoretically
possible problems is vastly larger than the areas where we see
problems in practice. You have to be pragmatic about it.

--
Peter Geoghegan

In reply to: David E. Wheeler (#12)
Re: Proposal: Document ABI Compatibility

On Tue, Jun 11, 2024 at 10:55 AM David E. Wheeler <david@justatheory.com> wrote:

Right, it’s just that extension authors could use some notification that such a change is coming so they can update their code, if necessary.

In general our strategy around ABI breaks is to avoid them whenever
possible. We also make the most conservative assumptions about what a
true ABI break is -- strictly speaking we can never be fully sure
about the impact of a theoretical/mechanical ABI break (we can only
make well educated guesses). My sense is that we're approaching having
the fewest possible real ABI breaks already -- we're already doing the
best we can. That doesn't seem like a useful area to focus on.

As an example, my bugfix commit 714780dc was apparently discussed by
Yurii Rashkovskii during his pgConf.dev talk. I was (and still am)
approaching 100% certainty that that wasn't a true ABI break.
Documenting this somewhere seems rather unappealing. Strictly speaking
I'm not 100% certain that this is a non-issue, but who benefits from
hearing my hand-wavy characterisation of why I believe it's a
non-issue? You might as well just look at an ABI change report
yourself.

Approximately 0% of all extensions actually use the struct in
question, and so obviously aren't affected. If anybody is using the
struct then it's merely very very likely that they aren't affected.
But why trust me here? After all, I can't even imagine why anybody
would want to use the struct in question. My hand-wavy speculation
about what it would look like if I was wrong about that is inherently
suspect, and probably just useless. Is it not?

That having been said, it would be useful if there was a community web
resource for this -- something akin to coverage.postgresql.org, but
with differential ABI breakage reports. You can see an example report
here:

/messages/by-id/CAH2-Wzm-W6hSn71sUkz0Rem=qDEU7TnFmc7_jG2DjrLFef_WKQ@mail.gmail.com

Theoretically anybody can do this themselves. In practice they don't.
So something as simple as providing automated reports about ABI
changes might well move the needle here.

--
Peter Geoghegan

#23Peter Eisentraut
peter@eisentraut.org
In reply to: Robert Haas (#17)
Re: Proposal: Document ABI Compatibility

On 12.06.24 16:47, Robert Haas wrote:

On Mon, Jun 3, 2024 at 3:39 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:

Me either. There are degrees of ABI compatibility

Exactly this!

What I think would be useful to document is our usual practices e.g.
adding new struct members at the end of structs, trying to avoid
changing public function signatures. If we document promises to
extension authors, I don't know how much difference that will make:
we'll probably end up needing to violate them at some point for one
reason or another. But if we document what committers should do, then
we might do better than we're now, because committers will be more
likely to do it right, and extension authors can also read those
instructions to understand what our practices are.

Fun fact: At the end of src/tools/RELEASE_CHANGES, there is some
guidance on how to maintain ABI compatibility in *libpq*. That used to
be a problem. We have come far since then.

But yes, a bit of documentation like that (maybe not in that file
though) would make sense.

#24David E. Wheeler
david@justatheory.com
In reply to: Andres Freund (#19)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 11:20, Andres Freund <andres@anarazel.de> wrote:

The PostgreSQL core team maintains two application binary interface (ABI) guarantees: one for major releases and one for minor releases.

I.e. for major versions it's "there is none"?

Is it? ISTM that there is the intention not to break things that don’t need to be broken, though that doesn’t rule out interface improvements.

In such cases the incompatible changes will be listed in the Release Notes.

I don't think we actually exhaustively list all of them.

Should they be? I can maybe see the argument not to for major releases. But I’ve also has the experience of a new failure on a major release and having to go find the reason for it and the fix, often requiring the attention of someone on a mailing list who might rather tap the “compatibility changes” sign in the latest change log. :-)

s/every/a reasonable/ or just s/every/an/

The padding case doesn't affect sizeof() fwiw.

I think there's too often not an alternative to using sizeof(), potentially
indirectly (via makeNode() or such. So this sounds a bit too general.

Is there some other way to avoid the issue? Or would constructor APIs need to be added to core?

Updated with your suggestions:

``` md

ABI Policy
==========

The PostgreSQL core team maintains two application binary interface (ABI) guarantees: one for major releases and one for minor releases.

Major Releases
--------------

Applications that use the PostgreSQL APIs must be compiled for each major release supported by the application. The inclusion of `PG_MODULE_MAGIC` ensures that code compiled for one major version will rejected by other major versions.

Furthermore, new releases may make API changes that require code changes. Use the `PG_VERSION_NUM` constant to adjust code in a backwards compatible way:

``` c
#if PG_VERSION_NUM >= 160000
#include "varatt.h"
#endif
```

PostgreSQL avoids unnecessary API changes in major releases, but usually ships a few necessary API changes, including deprecation, renaming, and argument variation. In such cases the incompatible changes will be listed in the Release Notes.

Minor Releases
--------------

PostgreSQL makes an effort to avoid both API and ABI breaks in minor releases. In general, an application compiled against any minor release will work with any other minor release, past or future.

When a change *is* required, PostgreSQL will choose the least invasive way possible, for example by squeezing a new field into padding space or appending it to the end of a struct. This sort of change should not impact dependent applications unless, they use `sizeof(the struct)` on a struct with an appended field, or create their own instances of such structs --- patterns best avoided.

In rare cases, however, even such non-invasive changes may be impractical or impossible. In such an event, the change will be documented in the Release Notes, and details on the issue will also be posted to [TBD; mail list? Blog post? News item?].

The project strongly recommends that developers adopt continuous integration testing at least for the latest minor release all major versions of Postgres they support.
```

Best,

David

#25David E. Wheeler
david@justatheory.com
In reply to: Peter Geoghegan (#21)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 11:30, Peter Geoghegan <pg@bowt.ie> wrote:

I'm a little surprised that we don't seem to have all that many
problems with ABI breakage, though. Although we theoretically have a
huge number of APIs that extension authors might choose to use, that
isn't really true in practical terms. The universe of theoretically
possible problems is vastly larger than the areas where we see
problems in practice. You have to be pragmatic about it.

Things go wrong far less often than one might fear! Given this relative stability, I think it’s reasonable to document what heretofore assumed the policy is so that the fears can largely be put to rest by clear expectations.

Best,

David

#26David E. Wheeler
david@justatheory.com
In reply to: Peter Geoghegan (#22)
Re: Proposal: Document ABI Compatibility

On Jun 12, 2024, at 11:57, Peter Geoghegan <pg@bowt.ie> wrote:

That having been said, it would be useful if there was a community web
resource for this -- something akin to coverage.postgresql.org, but
with differential ABI breakage reports. You can see an example report
here:

/messages/by-id/CAH2-Wzm-W6hSn71sUkz0Rem=qDEU7TnFmc7_jG2DjrLFef_WKQ@mail.gmail.com

Theoretically anybody can do this themselves. In practice they don't.
So something as simple as providing automated reports about ABI
changes might well move the needle here.

What would be required to make such a thing? Maybe it’d make a good Summer of Code project.

Best,

David

#27Andreas 'ads' Scherbaum
adsmail@wars-nicht.de
In reply to: David E. Wheeler (#3)
Re: Proposal: Document ABI Compatibility

On 03/06/2024 21:21, David E. Wheeler wrote:

On Jun 3, 2024, at 14:58, Andres Freund <andres@anarazel.de> wrote:

Hi,

Hello Andres.

Are there notes for the session?

Yes, but not posted yet. Here’s what Andreas 'ads' Scherbaum sent me for that bit of the conversation:

* Core is focused on core ABI stability
* David: No "statement of stability" in Core
* David/Jeremy/Tom: coding guidelines, style guidelines
* useful to have docs in core about what's stable and what's not, what you should compile against or not, and ABI guarantees
* Abigale: there are hooks, but no overall concept for extensions
* Tom: Peter Eisentraut is working on tests for extensions stability
* Jeremy: nothing is preventing people from installing incompatible versions

The full "discussion" is here:

https://wiki.postgresql.org/wiki/PGConf.dev_2024_Developer_Unconference#Improving_extensions_in_core

And the ABI discussion here:
https://wiki.postgresql.org/wiki/PGConf.dev_2024_Extension_Summit#ABI.2FAPI_discussion

--
Andreas 'ads' Scherbaum
German PostgreSQL User Group
European PostgreSQL User Group - Board of Directors
Volunteer Regional Contact, Germany - PostgreSQL Project

#28Peter Eisentraut
peter@eisentraut.org
In reply to: David E. Wheeler (#24)
Re: Proposal: Document ABI Compatibility

On 18.06.24 00:37, David E. Wheeler wrote:

ABI Policy
==========

The PostgreSQL core team maintains two application binary interface (ABI) guarantees: one for major releases and one for minor releases.

Major Releases
--------------

Applications that use the PostgreSQL APIs

This is probably a bit confusing. This might as well mean client
application code against libpq. Better something like "server plugin
code that uses the PostgreSQL server APIs".

must be compiled for each major release supported by the application. The inclusion of `PG_MODULE_MAGIC` ensures that code compiled for one major version will rejected by other major versions.

ok so far

Furthermore, new releases may make API changes that require code changes. Use the `PG_VERSION_NUM` constant to adjust code in a backwards compatible way:

``` c
#if PG_VERSION_NUM >= 160000
#include "varatt.h"
#endif
```

But now we're talking about API. That might be subject of another
document or another section in this one, but it seems confusing to mix
this with the ABI discussion.

PostgreSQL avoids unnecessary API changes in major releases, but usually ships a few necessary API changes, including deprecation, renaming, and argument variation.

Obviously, as a practical matter, there won't be random pointless
changes. But I wouldn't go as far as writing anything down about how
these APIs are developed.

In such cases the incompatible changes will be listed in the Release Notes.

I don't think anyone is signing up to do that.

Minor Releases
--------------

PostgreSQL makes an effort to avoid both API and ABI breaks in minor releases. In general, an application compiled against any minor release will work with any other minor release, past or future.

When a change *is* required, PostgreSQL will choose the least invasive way possible, for example by squeezing a new field into padding space or appending it to the end of a struct. This sort of change should not impact dependent applications unless, they use `sizeof(the struct)` on a struct with an appended field, or create their own instances of such structs --- patterns best avoided.

In rare cases, however, even such non-invasive changes may be impractical or impossible. In such an event, the change will be documented in the Release Notes, and details on the issue will also be posted to [TBD; mail list? Blog post? News item?].

I think one major problem besides actively avoiding or managing such
minor-version ABI breaks is some automated detection. Otherwise, this
just means "we try, but who knows".

#29Peter Eisentraut
peter@eisentraut.org
In reply to: David E. Wheeler (#26)
Re: Proposal: Document ABI Compatibility

On 18.06.24 00:40, David E. Wheeler wrote:

On Jun 12, 2024, at 11:57, Peter Geoghegan <pg@bowt.ie> wrote:

That having been said, it would be useful if there was a community web
resource for this -- something akin to coverage.postgresql.org, but
with differential ABI breakage reports. You can see an example report
here:

/messages/by-id/CAH2-Wzm-W6hSn71sUkz0Rem=qDEU7TnFmc7_jG2DjrLFef_WKQ@mail.gmail.com

Theoretically anybody can do this themselves. In practice they don't.
So something as simple as providing automated reports about ABI
changes might well move the needle here.

What would be required to make such a thing? Maybe it’d make a good Summer of Code project.

The above thread contains a lengthy discussion about what one could do.

#30Robert Haas
robertmhaas@gmail.com
In reply to: David E. Wheeler (#24)
Re: Proposal: Document ABI Compatibility

On Mon, Jun 17, 2024 at 6:38 PM David E. Wheeler <david@justatheory.com> wrote:

Is it? ISTM that there is the intention not to break things that don’t need to be broken, though that doesn’t rule out interface improvements.

I suppose that it's true that we try to avoid gratuitous breakage, but
I feel like it would be weird to document that.

Sometimes I go to a store and I see a sign that says "shoplifters will
be prosecuted." But I have yet to see a store with a sign that says
"people who appear to be doing absolutely nothing wrong will not be
prosecuted." If I did see such a sign, I would frankly be a little
concerned.

--
Robert Haas
EDB: http://www.enterprisedb.com

#31David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#29)
Re: Proposal: Document ABI Compatibility

On Jun 19, 2024, at 05:42, Peter Eisentraut <peter@eisentraut.org> wrote:

/messages/by-id/CAH2-Wzm-W6hSn71sUkz0Rem=qDEU7TnFmc7_jG2DjrLFef_WKQ@mail.gmail.com

Theoretically anybody can do this themselves. In practice they don't.
So something as simple as providing automated reports about ABI
changes might well move the needle here.

What would be required to make such a thing? Maybe it’d make a good Summer of Code project.

The above thread contains a lengthy discussion about what one could do.

I somehow missed that GSoC 2024 is already going with contributors. Making a mental note to add an item for 2025.

D

#32David E. Wheeler
david@justatheory.com
In reply to: Robert Haas (#30)
Re: Proposal: Document ABI Compatibility

On Jun 24, 2024, at 14:51, Robert Haas <robertmhaas@gmail.com> wrote:

I suppose that it's true that we try to avoid gratuitous breakage, but
I feel like it would be weird to document that.

I see how that can seem weird to a committer deeply familiar with the development process and how things happen. But people outside the -hackers bubble have very little idea. It’s fair to say it needn’t be a long statement for major versions: a single sentence such as “we try to avoid gratuitous breakage” is a perfectly reasonable framing. But I’d say, in the interest of completeness, it would be useful to document the policy for major release *as well as* minor releases.

Best,

David

#33David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#28)
Re: Proposal: Document ABI Compatibility

On Jun 19, 2024, at 05:41, Peter Eisentraut <peter@eisentraut.org> wrote:

This is probably a bit confusing. This might as well mean client application code against libpq. Better something like "server plugin code that uses the PostgreSQL server APIs".

That works.

But now we're talking about API. That might be subject of another document or another section in this one, but it seems confusing to mix this with the ABI discussion.

Hrm. They’re super closely-related in my mind, as an extension developer. I need to know both! I guess I’m taking of this policy as what I can expect may be changed (and how to adapt to it) and what won’t.

That said, I’m fine to remove the API stuff if there’s consensus objecting to it, to be defined in a separate policy (perhaps on the same doc page).

PostgreSQL avoids unnecessary API changes in major releases, but usually ships a few necessary API changes, including deprecation, renaming, and argument variation.

Obviously, as a practical matter, there won't be random pointless changes. But I wouldn't go as far as writing anything down about how these APIs are developed.

Fair enough, was trying to give some idea of the sorts of changes. Don’t have to include them.

In such cases the incompatible changes will be listed in the Release Notes.

I don't think anyone is signing up to do that.

It needn’t be comprehensive. Just mention that an ABI or API changed in the release note item. Unless they almost *all* make such changes.

Minor Releases
--------------

I think one major problem besides actively avoiding or managing such minor-version ABI breaks is some automated detection. Otherwise, this just means "we try, but who knows”.

I think you *do* try, and the fact that there are so few issues means you succeed at that. I’m not advocating for an ABI guarantee here, just a description of the policy committees already follow.

Here’s an update based on all the feedback, framing things more from the perspective of “do I need to recompile this or change my code”. Many thanks!

``` md
ABI Policy
==========

Changes to the the PostgreSQL server APIs may require recompilation of server plugin code that uses them. This policy describes the core team's approach to such changes, and what server API users can expect.

Major Releases
--------------

Applications that use server APIs must be compiled for each major release supported by the application. The inclusion of `PG_MODULE_MAGIC` ensures that code compiled for one major version will rejected by other major versions. Developers needing to support multiple versions of PostgreSQL with incompatible APIs should use the `PG_VERSION_NUM` constant to adjust code as appropriate. For example:

``` c
#if PG_VERSION_NUM >= 160000
#include "varatt.h"
#endif
```

The core team avoids unnecessary breakage, but users of the server APIs should expect and be prepared to make adjustments and recompile for every major release.

Minor Releases
--------------

PostgreSQL makes an effort to avoid server API and ABI breaks in minor releases. In general, an application compiled against any minor release will work with any other minor release, past or future. In the absence of automated detection of such changes, this is not a guarantee, but history such breaking changes have been extremely rare.

When a change *is* required, PostgreSQL will choose the least invasive change possible, for example by squeezing a new field into padding space or appending it to the end of a struct. This sort of change should not impact dependent applications unless they use `sizeof(the struct)` on a struct with an appended field, or create their own instances of such structs --- patterns best avoided.

In rare cases, however, even such non-invasive changes may be impractical or impossible. In such an event, the change will be documented in the Release Notes, and details on the issue will also be posted to [TBD; mail list? Blog post? News item?].

To minimize issues and catch changes early, the project strongly recommends that developers adopt continuous integration testing at least for the latest minor release all major versions of Postgres they support.
```

Best,

David

#34Peter Eisentraut
peter@eisentraut.org
In reply to: David E. Wheeler (#33)
Re: Proposal: Document ABI Compatibility

On 24.06.24 22:26, David E. Wheeler wrote:

But now we're talking about API. That might be subject of another document or another section in this one, but it seems confusing to mix this with the ABI discussion.

Hrm. They’re super closely-related in my mind, as an extension developer. I need to know both! I guess I’m taking of this policy as what I can expect may be changed (and how to adapt to it) and what won’t.

That said, I’m fine to remove the API stuff if there’s consensus objecting to it, to be defined in a separate policy (perhaps on the same doc page).

I took at a stab at this, using some of your text, but discussing API
and ABI separately.

# Server API and ABI guidance

This section contains guidance to authors of extensions and other
server plugins about ABI and API stability in the PostgreSQL server.

## General

The PostgreSQL server contains several well-delimited APIs for server
plugins, such as the function manager (fmgr), SPI, and various hooks
specifically designed for extensions. These interfaces are carefully
managed for long-term stability and compatibility. However, the
entire set of global functions and variables in the server effectively
constitutes the publicly usable API, but most parts of that were not
designed with extensibility and long-term stability in mind. That
means, while taking advantage of these interfaces is valid, the
further one strays from the well-trodden path, the likelier it will be
that one might encounter ABI or API compatibility issues at some
point. Extension authors are also encouraged to provide feedback
about their requirements, so that over time, as new use patterns
arise, certain interfaces can be consider more stabilized or new
better-designed interfaces for new uses can be added.

## API compatibility

(API = application programming interface, meaning the interface used
at compile time)

### Major versions

There is _no_ promise of API compatibility between PostgreSQL major
versions. That means, extension code might require source code
changes to work with multiple major versions. These can usually be
managed with preprocessor conditions like `#if PG_VERSION_NUM >=
160000`. Sophisticated extensions that use interfaces beyond the
well-delimited ones usually require a few such changes for each major
server version.

### Minor versions

PostgreSQL makes an effort to avoid server API breaks in minor
releases. In general, extension code that compiles and works with
some minor release should also compile and work with any other minor
release, past or future.

When a change *is* required, this will be carefully managed, taking
the requirements of extensions into account. Such changes will be
communicated in the release notes.

## ABI compatibility

(ABI = application binary interface, meaning the interface used at run
time)

### Major versions

Servers of different major versions have intentionally incompatible
ABIs. That means, extensions that use server APIs must be re-compiled
for each major release. The inclusion of `PG_MODULE_MAGIC` ensures
that code compiled for one major version will be rejected by other
major versions.

### Minor versions

PostgreSQL makes an effort to avoid server ABI breaks in minor
releases. In general, an extension compiled against any minor release
should work with any other minor release, past or future.

When a change *is* required, PostgreSQL will choose the least invasive
change possible, for example by squeezing a new field into padding
space or appending it to the end of a struct. These sorts of changes
should not impact extensions unless they use very unusual code
patterns.

In rare cases, however, even such non-invasive changes may be
impractical or impossible. In such an event, the change will be
carefully managed, taking the requirements of extensions into account.
Such changes will also be documented in the release notes.

Note, however, again that many parts of the server are not designed or
maintained as publicly-consumable APIs (and that, in most cases, the
actual boundary is also not well-defined). If urgent needs arise,
changes in those parts will naturally be done with less consideration
for extension code than changes in well-defined and widely used
interfaces.

Also, in the absence of automated detection of such changes, this is
not a guarantee, but historically such breaking changes have been
extremely rare.

In reply to: Peter Eisentraut (#34)
Re: Proposal: Document ABI Compatibility

Peter Eisentraut <peter@eisentraut.org> writes:

On 24.06.24 22:26, David E. Wheeler wrote:

But now we're talking about API. That might be subject of another

document or another section in this one, but it seems confusing to mix
this with the ABI discussion.
Hrm. They’re super closely-related in my mind, as an extension
developer. I need to know both! I guess I’m taking of this policy as
what I can expect may be changed (and how to adapt to it) and what
won’t.
That said, I’m fine to remove the API stuff if there’s consensus
objecting to it, to be defined in a separate policy (perhaps on the
same doc page).

I took at a stab at this, using some of your text, but discussing API
and ABI separately.

This looks good to me, just one minor nitpick:

### Minor versions

PostgreSQL makes an effort to avoid server API breaks in minor
releases. In general, extension code that compiles and works with
some minor release should also compile and work with any other minor
release, past or future.

I think this should explicitly say "any other minor release within [or
"from" or "of"?] the same major version" (and ditto in the ABI section).

- ilmari

#36David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#34)
Re: Proposal: Document ABI Compatibility

On Jun 25, 2024, at 7:33 AM, Peter Eisentraut <peter@eisentraut.org> wrote:

I took at a stab at this, using some of your text, but discussing API and ABI separately.

Oh man this is fantastic, thank you! I’d be more than happy to just turn this into a patch. But where should it go? Upthread I assumed xfunc.sgml, and still think that’s a likely candidate. Perhaps I’ll just start there --- unless someone thinks it should go somewhere other than the docs.

Best,

David

#37Laurenz Albe
laurenz.albe@cybertec.at
In reply to: David E. Wheeler (#36)
Re: Proposal: Document ABI Compatibility

On Tue, 2024-06-25 at 13:55 -0400, David E. Wheeler wrote:

On Jun 25, 2024, at 7:33 AM, Peter Eisentraut <peter@eisentraut.org> wrote:

I took at a stab at this, using some of your text, but discussing API and ABI separately.

Oh man this is fantastic, thank you! I’d be more than happy to just turn this into a patch.
But where should it go? Upthread I assumed xfunc.sgml, and still think that’s a likely
candidate. Perhaps I’ll just start there --- unless someone thinks it should go somewhere
other than the docs.

Perhaps such information should go somewhere here:
https://www.postgresql.org/support/versioning/

Yours,
Laurenz Albe

#38David E. Wheeler
david@justatheory.com
In reply to: Laurenz Albe (#37)
Re: Proposal: Document ABI Compatibility

On Jun 26, 2024, at 04:48, Laurenz Albe <laurenz.albe@cybertec.at> wrote:

Perhaps such information should go somewhere here:
https://www.postgresql.org/support/versioning/

This seems deeper and more detailed than what’s there now, but I can certainly imagine wanting to include this policy on the web site. That said, it didn’t occur to me to look under support when trying to find a place to put this; I was looking under Developers, on the principle that extension developers would look there.

Best,

David

#39David E. Wheeler
david@justatheory.com
In reply to: David E. Wheeler (#36)
1 attachment(s)
Re: Proposal: Document ABI Compatibility

On Jun 25, 2024, at 13:55, David E. Wheeler <david@justatheory.com> wrote:

Oh man this is fantastic, thank you! I’d be more than happy to just turn this into a patch. But where should it go? Upthread I assumed xfunc.sgml, and still think that’s a likely candidate. Perhaps I’ll just start there --- unless someone thinks it should go somewhere other than the docs.

Okay here’s a patch that adds the proposed API and ABI guidance to the C Language docs. The content is the same as Peter proposed, with some light copy-editing.

Best,

David

Attachments:

v1-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patchapplication/octet-stream; name=v1-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patch; x-unix-mode=0644Download
From 9b65ee342a9774922e96f9ddaae9f353de013266 Mon Sep 17 00:00:00 2001
From: "David E. Wheeler" <david@justatheory.com>
Date: Wed, 26 Jun 2024 15:07:56 -0400
Subject: [PATCH v1] Add API an ABI guidance to the C language docs

Includes guidance for major and minor version releases, and sets reasonable
expectations for extension developers to follow.

Author: David Wheeler, Peter Eisentraut

Discussion: https://www.postgresql.org/message-id/flat/5DA9F9D2-B8B2-43DE-BD4D-53A4160F6E8D%40justatheory.com
---
 doc/src/sgml/xfunc.sgml | 143 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 138 insertions(+), 5 deletions(-)

diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 1d0b65193e..71832db608 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -2704,15 +2704,148 @@ CREATE FUNCTION concat_text(text, text) RETURNS text
 
 &dfunc;
 
+   <sect2 id="xfunc-guidance">
+    <title>Server API and ABI Guidance</title>
+
+    <para>
+     This section contains guidance to authors of extensions and other server
+     plugins about ABI and API stability in the
+     <productname>PostgreSQL</productname> server.
+    </para>
+
+    <sect3 id="xfunc-guidance-general">
+     <title>General</title>
+
+     <para>
+      The <productname>PostgreSQL</productname> server contains several
+      well-delimited APIs for server plugins, such as the function manager
+      (<xref linkend="xfunc-c-polymorphic"><acronym>fmgr</acronym></xref>),
+      <xref linkend="spi"><acronym>SPI</acronym></xref>, and various hooks
+      specifically designed for extensions. These interfaces are carefully
+      managed for long-term stability and compatibility. However, the entire
+      set of global functions and variables in the server effectively
+      constitutes the publicly usable API, and most of it was not
+      designed with extensibility and long-term stability in mind.
+     </para>
+
+     <para>
+      Therefore, while taking advantage of these interfaces is valid, the
+      further one strays from the well-trodden path, the likelier it will be
+      that one might encounter ABI or API compatibility issues at some point.
+      Extension authors are encouraged to provide feedback about their
+      requirements, so that over time, as new use patterns arise, certain
+      interfaces can be considerd more stabilized or new, better-designed
+      interfaces can be added.
+     </para>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-api-compatibility">
+     <title>API Compatibility</title>
+     <para>
+      The <acronym>API</acronym> or application programming interface, is the
+      interface used at compile time.
+     </para>
+
+     <sect4 id="xfunc-guidance-api-major-versions">
+      <title>Major Versions</title>
+      <para>
+       There is <emphasis>no</emphasis> promise of API compatibility between
+       <productname>PostgreSQL</productname> major versions. Extension code
+       therefore might require source code changes to work with multiple major
+       versions. These can usually be managed with preprocessor conditions
+       such as <literal>#if PG_VERSION_NUM &gt;= 160000</literal>.
+       Sophisticated extensions that use interfaces beyond the well-defined
+       ones usually require a few such changes for each major server version.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-api-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       API breaks in minor releases. In general, extension code that compiles
+       and works with a minor release should also compile and work with any
+       other minor release of the same major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required, it will be carefully
+       managed, taking the requirements of extensions into account. Such
+       changes will be communicated in the <xref linkend="release"/>.
+      </para>
+     </sect4>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-abi-compatibility">
+     <title>ABI Compatibility</title>
+      <para>
+       The <acronym>ABI</acronym> or application binary interface, is the
+       interface used at run time.
+      </para>
+
+     <sect4 id="xfunc-guidance-abi-major-versions">
+      <title>Major Versions</title>
+      <para>
+       Servers of different major versions have intentionally incompatible
+       ABIs. Extensions that use server APIs must therefore be re-compiled for
+       each major release. The inclusion of <literal>PG_MODULE_MAGIC</literal>
+       ensures that code compiled for one major version will be rejected by
+       other major versions.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-abi-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       ABI breaks in minor releases. In general, an extension compiled against
+       any minor release should work with any other minor release of the same
+       major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required,
+       <productname>PostgreSQL</productname> will choose the least invasive
+       change possible, for example by squeezing a new field into padding
+       space or appending it to the end of a struct. These sorts of changes
+       should not impact extensions unless they use very unusual code
+       patterns.
+      </para>
+
+      <para>
+       In rare cases, however, even such non-invasive changes may be
+       impractical or impossible. In such an event, the change will be
+       carefully managed, taking the requirements of extensions into account.
+       Such changes will also be documented in the <xref linkend="release"/>.
+      </para>
+
+      <para>
+       Note, however, that many parts of the server are not designed or
+       maintained as publicly-consumable APIs (and that, in most cases, the
+       actual boundary is also not well-defined). If urgent needs arise,
+       changes in those parts will naturally be made with less consideration
+       for extension code than changes in well-defined and widely used
+       interfaces.
+      </para>
+
+      <para>
+       Also, in the absence of automated detection of such changes, this is
+       not a guarantee, but historically such breaking changes have been
+       extremely rare.
+      </para>
+
+     </sect4>
+    </sect3>
+  </sect2>
+
    <sect2 id="xfunc-c-composite-type-args">
     <title>Composite-Type Arguments</title>
 
     <para>
-     Composite types do not have a fixed layout like C structures.
-     Instances of a composite type can contain null fields.  In
-     addition, composite types that are part of an inheritance
-     hierarchy can have different fields than other members of the
-     same inheritance hierarchy.  Therefore,
+     Composite types do not have a fixed layout like C structures. Instances
+     of a composite type can contain null fields.  In addition, composite
+     types that are part of an inheritance hierarchy can have different fields
+     than other members of the same inheritance hierarchy. Therefore,
      <productname>PostgreSQL</productname> provides a function
      interface for accessing fields of composite types from C.
     </para>
-- 
2.45.2

#40David E. Wheeler
david@justatheory.com
In reply to: David E. Wheeler (#39)
Re: Proposal: Document ABI Compatibility

On Jun 26, 2024, at 15:14, David E. Wheeler <david@justatheory.com> wrote:

Okay here’s a patch that adds the proposed API and ABI guidance to the C Language docs. The content is the same as Peter proposed, with some light copy-editing.

CF: https://commitfest.postgresql.org/48/5080/
PR: https://github.com/theory/postgres/pull/6

D

#41David E. Wheeler
david@justatheory.com
In reply to: David E. Wheeler (#40)
1 attachment(s)
Re: Proposal: Document ABI Compatibility

On Jun 26, 2024, at 15:20, David E. Wheeler <david@justatheory.com> wrote:

CF: https://commitfest.postgresql.org/48/5080/
PR: https://github.com/theory/postgres/pull/6

Aaaand v2 without the unnecessary formatting of unrelated documentation 🤦🏻‍♂️.

Best,

David

Attachments:

v2-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patchapplication/octet-stream; name=v2-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patch; x-unix-mode=0644Download
From 12a48284babf2a0c9a939492e4d443be8fdcac30 Mon Sep 17 00:00:00 2001
From: "David E. Wheeler" <david@justatheory.com>
Date: Wed, 26 Jun 2024 15:22:07 -0400
Subject: [PATCH v2] Add API an ABI guidance to the C language docs

Includes guidance for major and minor version releases, and sets reasonable
expectations for extension developers to follow.

Author: David Wheeler, Peter Eisentraut

Discussion: https://www.postgresql.org/message-id/flat/5DA9F9D2-B8B2-43DE-BD4D-53A4160F6E8D%40justatheory.com
---
 doc/src/sgml/xfunc.sgml | 134 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)

diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 1d0b65193e..2721e70256 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -2704,6 +2704,140 @@ CREATE FUNCTION concat_text(text, text) RETURNS text
 
 &dfunc;
 
+   <sect2 id="xfunc-guidance">
+    <title>Server API and ABI Guidance</title>
+
+    <para>
+     This section contains guidance to authors of extensions and other server
+     plugins about ABI and API stability in the
+     <productname>PostgreSQL</productname> server.
+    </para>
+
+    <sect3 id="xfunc-guidance-general">
+     <title>General</title>
+
+     <para>
+      The <productname>PostgreSQL</productname> server contains several
+      well-delimited APIs for server plugins, such as the function manager
+      (<xref linkend="xfunc-c-polymorphic"><acronym>fmgr</acronym></xref>),
+      <xref linkend="spi"><acronym>SPI</acronym></xref>, and various hooks
+      specifically designed for extensions. These interfaces are carefully
+      managed for long-term stability and compatibility. However, the entire
+      set of global functions and variables in the server effectively
+      constitutes the publicly usable API, and most of it was not
+      designed with extensibility and long-term stability in mind.
+     </para>
+
+     <para>
+      Therefore, while taking advantage of these interfaces is valid, the
+      further one strays from the well-trodden path, the likelier it will be
+      that one might encounter ABI or API compatibility issues at some point.
+      Extension authors are encouraged to provide feedback about their
+      requirements, so that over time, as new use patterns arise, certain
+      interfaces can be considerd more stabilized or new, better-designed
+      interfaces can be added.
+     </para>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-api-compatibility">
+     <title>API Compatibility</title>
+     <para>
+      The <acronym>API</acronym> or application programming interface, is the
+      interface used at compile time.
+     </para>
+
+     <sect4 id="xfunc-guidance-api-major-versions">
+      <title>Major Versions</title>
+      <para>
+       There is <emphasis>no</emphasis> promise of API compatibility between
+       <productname>PostgreSQL</productname> major versions. Extension code
+       therefore might require source code changes to work with multiple major
+       versions. These can usually be managed with preprocessor conditions
+       such as <literal>#if PG_VERSION_NUM &gt;= 160000</literal>.
+       Sophisticated extensions that use interfaces beyond the well-defined
+       ones usually require a few such changes for each major server version.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-api-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       API breaks in minor releases. In general, extension code that compiles
+       and works with a minor release should also compile and work with any
+       other minor release of the same major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required, it will be carefully
+       managed, taking the requirements of extensions into account. Such
+       changes will be communicated in the <xref linkend="release"/>.
+      </para>
+     </sect4>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-abi-compatibility">
+     <title>ABI Compatibility</title>
+      <para>
+       The <acronym>ABI</acronym> or application binary interface, is the
+       interface used at run time.
+      </para>
+
+     <sect4 id="xfunc-guidance-abi-major-versions">
+      <title>Major Versions</title>
+      <para>
+       Servers of different major versions have intentionally incompatible
+       ABIs. Extensions that use server APIs must therefore be re-compiled for
+       each major release. The inclusion of <literal>PG_MODULE_MAGIC</literal>
+       ensures that code compiled for one major version will be rejected by
+       other major versions.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-abi-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       ABI breaks in minor releases. In general, an extension compiled against
+       any minor release should work with any other minor release of the same
+       major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required,
+       <productname>PostgreSQL</productname> will choose the least invasive
+       change possible, for example by squeezing a new field into padding
+       space or appending it to the end of a struct. These sorts of changes
+       should not impact extensions unless they use very unusual code
+       patterns.
+      </para>
+
+      <para>
+       In rare cases, however, even such non-invasive changes may be
+       impractical or impossible. In such an event, the change will be
+       carefully managed, taking the requirements of extensions into account.
+       Such changes will also be documented in the <xref linkend="release"/>.
+      </para>
+
+      <para>
+       Note, however, that many parts of the server are not designed or
+       maintained as publicly-consumable APIs (and that, in most cases, the
+       actual boundary is also not well-defined). If urgent needs arise,
+       changes in those parts will naturally be made with less consideration
+       for extension code than changes in well-defined and widely used
+       interfaces.
+      </para>
+
+      <para>
+       Also, in the absence of automated detection of such changes, this is
+       not a guarantee, but historically such breaking changes have been
+       extremely rare.
+      </para>
+
+     </sect4>
+    </sect3>
+  </sect2>
+
    <sect2 id="xfunc-c-composite-type-args">
     <title>Composite-Type Arguments</title>
 
-- 
2.45.2

#42Jeremy Schneider
schneider@ardentperf.com
In reply to: David E. Wheeler (#41)
Re: Proposal: Document ABI Compatibility

On 6/26/24 12:23 PM, David E. Wheeler wrote:

On Jun 26, 2024, at 15:20, David E. Wheeler <david@justatheory.com> wrote:

CF: https://commitfest.postgresql.org/48/5080/
PR: https://github.com/theory/postgres/pull/6

Aaaand v2 without the unnecessary formatting of unrelated documentation 🤦🏻‍♂️.

Minor nit - misspelled "considerd"

-Jeremy

--
http://about.me/jeremy_schneider

#43David E. Wheeler
david@justatheory.com
In reply to: Jeremy Schneider (#42)
1 attachment(s)
Re: Proposal: Document ABI Compatibility

On Jun 27, 2024, at 17:48, Jeremy Schneider <schneider@ardentperf.com> wrote:

Minor nit - misspelled “considerd"

Thank you, Jeremy. V3 attached.

Best,

David

Attachments:

v3-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patchapplication/octet-stream; name=v3-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patch; x-unix-mode=0644Download
From 036bd1e0d6b36c8a1ee28bae5665b0f2ea1cb2d0 Mon Sep 17 00:00:00 2001
From: "David E. Wheeler" <david@justatheory.com>
Date: Thu, 27 Jun 2024 18:06:09 -0400
Subject: [PATCH v3] Add API an ABI guidance to the C language docs

Includes guidance for major and minor version releases, and sets reasonable
expectations for extension developers to follow.

Author: David Wheeler, Peter Eisentraut

Discussion: https://www.postgresql.org/message-id/flat/5DA9F9D2-B8B2-43DE-BD4D-53A4160F6E8D%40justatheory.com
---
 doc/src/sgml/xfunc.sgml | 134 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)

diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 1d0b65193e..e4a46e1c9c 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -2704,6 +2704,140 @@ CREATE FUNCTION concat_text(text, text) RETURNS text
 
 &dfunc;
 
+   <sect2 id="xfunc-guidance">
+    <title>Server API and ABI Guidance</title>
+
+    <para>
+     This section contains guidance to authors of extensions and other server
+     plugins about ABI and API stability in the
+     <productname>PostgreSQL</productname> server.
+    </para>
+
+    <sect3 id="xfunc-guidance-general">
+     <title>General</title>
+
+     <para>
+      The <productname>PostgreSQL</productname> server contains several
+      well-delimited APIs for server plugins, such as the function manager
+      (<xref linkend="xfunc-c-polymorphic"><acronym>fmgr</acronym></xref>),
+      <xref linkend="spi"><acronym>SPI</acronym></xref>, and various hooks
+      specifically designed for extensions. These interfaces are carefully
+      managed for long-term stability and compatibility. However, the entire
+      set of global functions and variables in the server effectively
+      constitutes the publicly usable API, and most of it was not
+      designed with extensibility and long-term stability in mind.
+     </para>
+
+     <para>
+      Therefore, while taking advantage of these interfaces is valid, the
+      further one strays from the well-trodden path, the likelier it will be
+      that one might encounter ABI or API compatibility issues at some point.
+      Extension authors are encouraged to provide feedback about their
+      requirements, so that over time, as new use patterns arise, certain
+      interfaces can be considered more stabilized or new, better-designed
+      interfaces can be added.
+     </para>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-api-compatibility">
+     <title>API Compatibility</title>
+     <para>
+      The <acronym>API</acronym> or application programming interface, is the
+      interface used at compile time.
+     </para>
+
+     <sect4 id="xfunc-guidance-api-major-versions">
+      <title>Major Versions</title>
+      <para>
+       There is <emphasis>no</emphasis> promise of API compatibility between
+       <productname>PostgreSQL</productname> major versions. Extension code
+       therefore might require source code changes to work with multiple major
+       versions. These can usually be managed with preprocessor conditions
+       such as <literal>#if PG_VERSION_NUM &gt;= 160000</literal>.
+       Sophisticated extensions that use interfaces beyond the well-defined
+       ones usually require a few such changes for each major server version.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-api-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       API breaks in minor releases. In general, extension code that compiles
+       and works with a minor release should also compile and work with any
+       other minor release of the same major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required, it will be carefully
+       managed, taking the requirements of extensions into account. Such
+       changes will be communicated in the <xref linkend="release"/>.
+      </para>
+     </sect4>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-abi-compatibility">
+     <title>ABI Compatibility</title>
+      <para>
+       The <acronym>ABI</acronym> or application binary interface, is the
+       interface used at run time.
+      </para>
+
+     <sect4 id="xfunc-guidance-abi-major-versions">
+      <title>Major Versions</title>
+      <para>
+       Servers of different major versions have intentionally incompatible
+       ABIs. Extensions that use server APIs must therefore be re-compiled for
+       each major release. The inclusion of <literal>PG_MODULE_MAGIC</literal>
+       ensures that code compiled for one major version will be rejected by
+       other major versions.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-abi-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       ABI breaks in minor releases. In general, an extension compiled against
+       any minor release should work with any other minor release of the same
+       major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required,
+       <productname>PostgreSQL</productname> will choose the least invasive
+       change possible, for example by squeezing a new field into padding
+       space or appending it to the end of a struct. These sorts of changes
+       should not impact extensions unless they use very unusual code
+       patterns.
+      </para>
+
+      <para>
+       In rare cases, however, even such non-invasive changes may be
+       impractical or impossible. In such an event, the change will be
+       carefully managed, taking the requirements of extensions into account.
+       Such changes will also be documented in the <xref linkend="release"/>.
+      </para>
+
+      <para>
+       Note, however, that many parts of the server are not designed or
+       maintained as publicly-consumable APIs (and that, in most cases, the
+       actual boundary is also not well-defined). If urgent needs arise,
+       changes in those parts will naturally be made with less consideration
+       for extension code than changes in well-defined and widely used
+       interfaces.
+      </para>
+
+      <para>
+       Also, in the absence of automated detection of such changes, this is
+       not a guarantee, but historically such breaking changes have been
+       extremely rare.
+      </para>
+
+     </sect4>
+    </sect3>
+  </sect2>
+
    <sect2 id="xfunc-c-composite-type-args">
     <title>Composite-Type Arguments</title>
 
-- 
2.45.2

#44David E. Wheeler
david@justatheory.com
In reply to: David E. Wheeler (#43)
1 attachment(s)
Re: Proposal: Document ABI Compatibility

On Jun 27, 2024, at 18:07, David E. Wheeler <david@justatheory.com> wrote:

Minor nit - misspelled “considerd"

Thank you, Jeremy. V3 attached.

Rebase on 5784a49 attached. I presume this topic needs quite a bit of review and consensus from the committers more generally.

Best,

David

Attachments:

v4-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patchapplication/octet-stream; name=v4-0001-Add-API-an-ABI-guidance-to-the-C-language-docs.patch; x-unix-mode=0644Download
From 7921cda4e4314211535bca17f92a8ede754577de Mon Sep 17 00:00:00 2001
From: "David E. Wheeler" <david@justatheory.com>
Date: Thu, 27 Jun 2024 18:06:09 -0400
Subject: [PATCH v4] Add API an ABI guidance to the C language docs

Includes guidance for major and minor version releases, and sets reasonable
expectations for extension developers to follow.

Author: David Wheeler, Peter Eisentraut

Discussion: https://www.postgresql.org/message-id/flat/5DA9F9D2-B8B2-43DE-BD4D-53A4160F6E8D%40justatheory.com
---
 doc/src/sgml/xfunc.sgml | 134 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 134 insertions(+)

diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index 7e92e89846..7e4e3ea4ae 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -2704,6 +2704,140 @@ CREATE FUNCTION concat_text(text, text) RETURNS text
 
 &dfunc;
 
+   <sect2 id="xfunc-guidance">
+    <title>Server API and ABI Guidance</title>
+
+    <para>
+     This section contains guidance to authors of extensions and other server
+     plugins about ABI and API stability in the
+     <productname>PostgreSQL</productname> server.
+    </para>
+
+    <sect3 id="xfunc-guidance-general">
+     <title>General</title>
+
+     <para>
+      The <productname>PostgreSQL</productname> server contains several
+      well-delimited APIs for server plugins, such as the function manager
+      (<xref linkend="xfunc-c-polymorphic"><acronym>fmgr</acronym></xref>),
+      <xref linkend="spi"><acronym>SPI</acronym></xref>, and various hooks
+      specifically designed for extensions. These interfaces are carefully
+      managed for long-term stability and compatibility. However, the entire
+      set of global functions and variables in the server effectively
+      constitutes the publicly usable API, and most of it was not
+      designed with extensibility and long-term stability in mind.
+     </para>
+
+     <para>
+      Therefore, while taking advantage of these interfaces is valid, the
+      further one strays from the well-trodden path, the likelier it will be
+      that one might encounter ABI or API compatibility issues at some point.
+      Extension authors are encouraged to provide feedback about their
+      requirements, so that over time, as new use patterns arise, certain
+      interfaces can be considered more stabilized or new, better-designed
+      interfaces can be added.
+     </para>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-api-compatibility">
+     <title>API Compatibility</title>
+     <para>
+      The <acronym>API</acronym> or application programming interface, is the
+      interface used at compile time.
+     </para>
+
+     <sect4 id="xfunc-guidance-api-major-versions">
+      <title>Major Versions</title>
+      <para>
+       There is <emphasis>no</emphasis> promise of API compatibility between
+       <productname>PostgreSQL</productname> major versions. Extension code
+       therefore might require source code changes to work with multiple major
+       versions. These can usually be managed with preprocessor conditions
+       such as <literal>#if PG_VERSION_NUM &gt;= 160000</literal>.
+       Sophisticated extensions that use interfaces beyond the well-defined
+       ones usually require a few such changes for each major server version.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-api-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       API breaks in minor releases. In general, extension code that compiles
+       and works with a minor release should also compile and work with any
+       other minor release of the same major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required, it will be carefully
+       managed, taking the requirements of extensions into account. Such
+       changes will be communicated in the <xref linkend="release"/>.
+      </para>
+     </sect4>
+    </sect3>
+
+    <sect3 id="xfunc-guidance-abi-compatibility">
+     <title>ABI Compatibility</title>
+      <para>
+       The <acronym>ABI</acronym> or application binary interface, is the
+       interface used at run time.
+      </para>
+
+     <sect4 id="xfunc-guidance-abi-major-versions">
+      <title>Major Versions</title>
+      <para>
+       Servers of different major versions have intentionally incompatible
+       ABIs. Extensions that use server APIs must therefore be re-compiled for
+       each major release. The inclusion of <literal>PG_MODULE_MAGIC</literal>
+       ensures that code compiled for one major version will be rejected by
+       other major versions.
+      </para>
+     </sect4>
+
+     <sect4 id="xfunc-guidance-abi-mninor-versions">
+      <title>Minor Versions</title>
+      <para>
+       <productname>PostgreSQL</productname> makes an effort to avoid server
+       ABI breaks in minor releases. In general, an extension compiled against
+       any minor release should work with any other minor release of the same
+       major version, past or future.
+      </para>
+
+      <para>
+       When a change <emphasis>is</emphasis> required,
+       <productname>PostgreSQL</productname> will choose the least invasive
+       change possible, for example by squeezing a new field into padding
+       space or appending it to the end of a struct. These sorts of changes
+       should not impact extensions unless they use very unusual code
+       patterns.
+      </para>
+
+      <para>
+       In rare cases, however, even such non-invasive changes may be
+       impractical or impossible. In such an event, the change will be
+       carefully managed, taking the requirements of extensions into account.
+       Such changes will also be documented in the <xref linkend="release"/>.
+      </para>
+
+      <para>
+       Note, however, that many parts of the server are not designed or
+       maintained as publicly-consumable APIs (and that, in most cases, the
+       actual boundary is also not well-defined). If urgent needs arise,
+       changes in those parts will naturally be made with less consideration
+       for extension code than changes in well-defined and widely used
+       interfaces.
+      </para>
+
+      <para>
+       Also, in the absence of automated detection of such changes, this is
+       not a guarantee, but historically such breaking changes have been
+       extremely rare.
+      </para>
+
+     </sect4>
+    </sect3>
+  </sect2>
+
    <sect2 id="xfunc-c-composite-type-args">
     <title>Composite-Type Arguments</title>
 
-- 
2.45.2

#45Peter Eisentraut
peter@eisentraut.org
In reply to: David E. Wheeler (#44)
Re: Proposal: Document ABI Compatibility

On 19.07.24 16:10, David E. Wheeler wrote:

On Jun 27, 2024, at 18:07, David E. Wheeler <david@justatheory.com> wrote:

Minor nit - misspelled “considerd"

Thank you, Jeremy. V3 attached.

Rebase on 5784a49 attached. I presume this topic needs quite a bit of review and consensus from the committers more generally.

Well, nobody has protested against what we wrote, so I have committed it.

#46David E. Wheeler
david@justatheory.com
In reply to: Peter Eisentraut (#45)
Re: Proposal: Document ABI Compatibility

On Jul 31, 2024, at 05:27, Peter Eisentraut <peter@eisentraut.org> wrote:

Well, nobody has protested against what we wrote, so I have committed it.

Excellent, thank you!

D