PoC: true in-place major version upgrades

pg_upgrade has worked for many years, but it still has well-known
limitations, in particular:
- Does not scale with big schemas, leading to hours of downtime
- No PITR across major versions

I think it's time for some innovation. The attached series is a
"tracer bullet", _rough_, proof of concept of in-place upgrade. With
the exception of 0001/2, the patches are not quite ready for code
review, but it's far enough along to demonstrate and get feedback on
architecture and orchestration. An earlier prototype showed this
should be much faster for big-schema databases, but it's not fair to
measure yet since it's not fully functional.

I have fleshed out a start-to-finish demo in the attached v1-demo.txt.
To give you a preview, the orchestration basically goes like this
(prepare, swap, golive):

$NEW/pg_inplace_upgrade -D $DEMO/primary -b $OLD -B $NEW \
--standby 'host=/tmp port=6001 dbname=postgres' prepare

$NEW/pg_inplace_upgrade -D $DEMO/primary -b $OLD -B $NEW -l
$DEMO/primary.log swap

$NEW/pg_inplace_upgrade -D $DEMO/primary -b $OLD -B $NEW golive

$NEW/pg_inplace_upgrade -D $DEMO/standby -b $OLD -B $NEW -l
$DEMO/standby.log swap

Here's what it does in a nutshell:

The 'prepare' stage runs some checks, freezes the catalogs, and shuts
down. The shutdown checkpoint is flagged as an upgrade boundary. The
standby will replay up to the primary's shutdown checkpoint and pause.

The 'swap' stages start the new binary. On the standby, it first shuts
down the old binary. On the primary, the startup process 'takes over'
the server by converting pg_control to the new format and writing a
start checkpoint. In a new 'upgrade' mode, a postmaster child migrates
the catalogs. (In the PoC, the catalog layout and data don't actually
change, so this simply rewrites the old catalogs' rows unchanged into
new files. It builds new indexes, and arranges for the relmapper and
pg_class to find the new files.) The catalog writes are WAL-logged,
but WAL archiving, sending, and recycling are suppressed for the
duration.

Up until 'golive', rollback is possible. This entails unlinking newly
created files (including catalog relations and pg_xact segments) and
restoring e.g. the control file and relmapper.

The 'golive' stage writes a "go-live" checkpoint, and resumes normal
WAL streaming and archiving. Rollback is now impossible.

The 'swap' stage on the standby allows it to replay the new-version
WAL generated during the migration, so the standby's catalog migration
is just applying normal FPIs. After replay passes the go-live
checkpoint, the standby can accept connections again.

*** Catalogs (TODO)

While the PoC sidesteps this entirely for now, this is actually how I
started. About a year and half ago, I experimented with catalog header
annotations so that the usual couple hundred lines of Perl would
generate the code that transforms the catalog tuples. Earlier this
year, I turned that into a throwaway "offline upgrade" (i.e. no WAL)
prototype. That exercise indicated that this method could feasibly
handle all catalog schema changes since PG12. Example from pg_class:

```
/* # of tuples (not always up-to-date; -1 means "unknown") */
float4 reltuples BKI_DEFAULT(-1) CHANGEDVALUE(14,
FUNC(reltuples_zero_to_unknown));

[...]
/* # of all-frozen blocks (not always up-to-date) */
int32 relallfrozen BKI_DEFAULT(0) FIRSTVERSION(18, CONST(0));
```

This says:
- "when upgrading from before PG14, call this function", which changes
'0' to '-1', otherwise takes reltuples as-is.
- "when generating tuple descriptors, leave out relallfrozen for
versions before PG18, and fill in a '0' for the new tuples"

There were some more difficult cases, but not many. The most difficult
aspect is node trees. I don't think it's possible to transform those
on-the-fly. My offline prototype dumped the deparsed node trees into
SQL in the usual way, which were then "recompiled" to node trees into
the new version's catalogs.

It should be possible for the catalog migration to bring the planner
statistics over as a matter of course.

*** Next steps

In order for in-place upgrade to actually work as a feature,

1) the old version needs to have 0001/2, so in practice that means the
upgraded-from version is PG20 at the earliest, with the rest of the
effort for PG21+.
2) At some point, catalog OIDs (e.g. between 10k and 12k) that now
vary between versions should be made immutable. Technically, it's not
necessary, but it makes it a lot nicer.

To justify committing those first two pieces during the PG20 cycle,
I'll need to progress the entire series to the point where there are
no serious architectural questions, and can be plausibly finished in
PG21+ with "just work". That will be a challenge, but I think it's
possible.

*** AI Disclaimer
I have used Claude extensively in the course of developing this. I was
skeptical about use of agents in software development until fairly
recently, but when given the right adult supervision they can be a
*huge* aid for complex software projects. I've re-written parts by
hand, but there's a lot more to clean up and harden. There are
operational gotchas and flaws, some of them left there on-purpose --
making this prematurely robust would be distracting and not a good use
of time before the architecture settles. The more boilerplate-y parts
have not had much scrutiny at all, but the important functionality
has.

--
John Naylor
Amazon Web Services