Prevent premature startup of pg_upgrade targets
Hackorum builds and tests every patch posted to the lists, not only commitfest submissions. This is Hackorum's own CI rather than the PostgreSQL project's, and it is still under testing - please report anything that looks wrong.
You can run a PostgreSQL built from this patch straight from Docker, with no checkout and no build:
docker run --rm -p 5432:5432 ghcr.io/hackorum-dev/postgres-patch:t253409psql -h localhost -U postgresBuilt from patchset v4 (message #4), October 14, 2026 at 12:08 PM.
Every patchset is also pushed to a branch of our PostgreSQL fork, so you can check out the same tree CI built. Without a PostgreSQL checkout:
git clone --branch t253409_4 https://github.com/hackorum-dev/postgres.gitIn a checkout you already have, add the fork once:
git remote add hackorum https://github.com/hackorum-dev/postgres.gitthen, for this patchset and every later one:
git fetch hackorum t253409_4 && git checkout t253409_4Patchset v4 (message #4) is on t253409_4
Hi hackers,
While investigating several corruption cases after major-version
upgrades, I noticed that a pg_upgrade target can be started normally
before pg_upgrade finishes, regardless of transfer mode.
On current master, setup() checks postmaster.pid for both clusters,
and get_control_data() checks for a clean shutdown. Both are initial
checks. There is no startup interlock during the upgrade.
pg_upgrade later starts and stops its own postmasters, so an external
startup may make a later start fail. But offline phases such as file
transfer, pg_resetwal, and syncing remain unprotected.
A premature startup allows clients, autovacuum, and background workers
to write WAL and advance xid and MultiXact state while pg_upgrade is
changing the cluster. pg_upgrade may still complete successfully.
In one production case, pg_upgrade completed successfully in copy
mode, and corruption was noticed weeks later. We observed missing
pg_xact state for tuple xmin, missing TOAST chunks, visibility map
inconsistencies, and VACUUM failing with “uncommitted xmin ... needs
to be frozen”. We also found evidence that the target had been started
during the upgrade. We have not proven that this caused the
corruption, but allowing such a startup looks unsafe. Refusing it
would make pg_upgrade more robust against automation errors.
There is already narrower protection for the old cluster. In link and
swap modes, disable_old_cluster() renames pg_control before file
transfer. The target has no equivalent protection.
I propose a durable marker in the target data directory that prevents
normal startup. Postmasters started by pg_upgrade would bypass it. The
marker would be removed and fsynced only when startup is safe.
For standby upgrades using rsync, the marker may need to remain after
pg_upgrade completes and be removed explicitly after rsync.
Some related discussions:
- old-cluster protection:
/messages/by-id/20120228162106.GA20764@momjian.us
- writes during binary upgrade:
/messages/by-id/20210121152357.s6eflhqyh4g5e6dv@dalibo.com
- logical replication launcher during pg_upgrade:
/messages/by-id/E1qyQMD-003XXN-Mf@gemulon.postgresql.org
- standby upgrades using rsync:
/messages/by-id/CAM527d8heqkjG5VrvjU3Xjsqxg41ufUyabD9QZccdAxnpbRH-Q@mail.gmail.com
What do you think?
regards,
Nik
On 14 Aug 2026, at 07:31, Nikolay Samokhvalov <nik@postgres.ai> wrote:
There is already narrower protection for the old cluster. In link and
swap modes, disable_old_cluster() renames pg_control before file
transfer. The target has no equivalent protection.
Sounds like a reasonable idea to add a layer of protection to the new cluster
as well, to prevent accidental startups.
--
Daniel Gustafsson
Hi Nik, Daniel,
I was able to reproduce this locally (16 -> 17, copy mode): I suspended
pg_upgrade during "Copying user relation files", started the target
cluster with pg_ctl, created a table and stopped it again. After
resuming, pg_upgrade reported "Upgrade Complete", and the table
created by the external server was present in the upgraded cluster.
Unless you're already working on this, Nik, I'd like to take it on as
my first patch. My rough plan:
- pg_upgrade creates a marker file in the new data directory (fsynced
together with the directory) before it first modifies the new
cluster, and removes it only after the final sync.
- The postmaster refuses to start while the marker exists, unless it
runs in binary-upgrade mode (-b), which pg_upgrade uses for its own
server starts.
- If pg_upgrade fails partway, the marker stays in place.
- TAP test and documentation updates.
I'd leave the rsync/standby case for a follow-up to keep the first
version small.
Two questions:
1. Should single-user mode also be blocked unless -b is given?
2. Any preference for the marker file name?
Regards,
Shubhra
[image: Shubhra Jain]
Shubhra Jain
Junior Software Engineer
[image: Phone] (+91) 9752683649 <(+91)+9752683649> [image: Website]
www.ksolves.com [image: Ksolves - AI First, Always]
On Fri, Oct 9, 2026 at 4:36 PM Nikolay Samokhvalov <nik@postgres.ai> wrote:
Show quoted text
Hi hackers,
While investigating several corruption cases after major-version
upgrades, I noticed that a pg_upgrade target can be started normally
before pg_upgrade finishes, regardless of transfer mode.On current master, setup() checks postmaster.pid for both clusters,
and get_control_data() checks for a clean shutdown. Both are initial
checks. There is no startup interlock during the upgrade.pg_upgrade later starts and stops its own postmasters, so an external
startup may make a later start fail. But offline phases such as file
transfer, pg_resetwal, and syncing remain unprotected.A premature startup allows clients, autovacuum, and background workers
to write WAL and advance xid and MultiXact state while pg_upgrade is
changing the cluster. pg_upgrade may still complete successfully.In one production case, pg_upgrade completed successfully in copy
mode, and corruption was noticed weeks later. We observed missing
pg_xact state for tuple xmin, missing TOAST chunks, visibility map
inconsistencies, and VACUUM failing with “uncommitted xmin ... needs
to be frozen”. We also found evidence that the target had been started
during the upgrade. We have not proven that this caused the
corruption, but allowing such a startup looks unsafe. Refusing it
would make pg_upgrade more robust against automation errors.There is already narrower protection for the old cluster. In link and
swap modes, disable_old_cluster() renames pg_control before file
transfer. The target has no equivalent protection.I propose a durable marker in the target data directory that prevents
normal startup. Postmasters started by pg_upgrade would bypass it. The
marker would be removed and fsynced only when startup is safe.For standby upgrades using rsync, the marker may need to remain after
pg_upgrade completes and be removed explicitly after rsync.Some related discussions:
- old-cluster protection:/messages/by-id/20120228162106.GA20764@momjian.us
- writes during binary upgrade:/messages/by-id/20210121152357.s6eflhqyh4g5e6dv@dalibo.com
- logical replication launcher during pg_upgrade:/messages/by-id/E1qyQMD-003XXN-Mf@gemulon.postgresql.org
- standby upgrades using rsync:/messages/by-id/CAM527d8heqkjG5VrvjU3Xjsqxg41ufUyabD9QZccdAxnpbRH-Q@mail.gmail.com
What do you think?
regards,
Nik
Hi,
Here is a first version of the patch.
pg_upgrade now creates a file named pg_upgrade_in_progress in the new
cluster's data directory right after the consistency checks, before it
first modifies the new cluster (so --check never creates it), and
removes it after the last step that starts the new cluster. Both the
creation and the removal are fsynced.
checkDataDir() refuses to start the server while the file exists,
unless IsBinaryUpgrade is set. pg_upgrade always starts its servers
with -b, so its own startups are unaffected. Doing the check in
checkDataDir() means it runs before postmaster.pid is created or
anything else is written, and it also covers single-user mode.
If pg_upgrade fails or is interrupted, the file stays in place and a
startup attempt fails with:
FATAL: data directory "..." is being upgraded by pg_upgrade
DETAIL: File ".../pg_upgrade_in_progress" exists.
HINT: Wait for pg_upgrade to complete. If pg_upgrade failed, the new
cluster must be re-created with initdb.
Testing:
- The reproduction from my previous mail (suspending pg_upgrade during
"Copying user relation files" and starting the target with pg_ctl)
now fails with the error above, and the upgrade completes normally
afterwards.
- Killing pg_upgrade during file transfer leaves the marker in place,
and the new cluster cannot be started.
- 002_pg_upgrade.pl now checks that --check does not create the marker,
that a successful upgrade removes it, and that the server refuses to
start while it exists.
- check-world passes, including all transfer modes in
006_transfer_modes.pl.
Open points:
1. The marker is fsynced even with --no-sync, since it is cheap and
safety-related. Happy to change that if preferred.
2. The #define lives in access/xlog_internal.h next to
XLOG_CONTROL_FILE, since it has to be usable from frontend code.
The file name itself is of course open for discussion.
3. The rsync/standby case Nik mentioned is not handled; keeping the
marker after completion could be a follow-up option.
Regards,
Shubhra