Patch for migration of the pg_commit_ts directory
Attachments:
v1-0001-Migration-of-the-pg_commit_ts-directory.patchtext/x-diff; name=v1-0001-Migration-of-the-pg_commit_ts-directory.patchDownload
From 5fcbacd870027afe9d59bec696fadfae5727db5b Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Thu, 3 Apr 2025 21:44:01 +0500
Subject: [PATCH v1] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/controldata.c | 23 ++++++++++++++++++++++-
src/bin/pg_upgrade/pg_upgrade.c | 6 ++++--
src/bin/pg_upgrade/pg_upgrade.h | 2 ++
3 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 47ee27ec835..0da5ee4225e 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -207,7 +207,8 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.data_checksum_version = 0;
got_data_checksum_version = true;
}
-
+ cluster->controldata.chkpnt_oldstCommitTsxid = 0;
+ cluster->controldata.chkpnt_newstCommitTsxid = 0;
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -320,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 9295e46aed3..746f8a6b02b 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -751,6 +751,8 @@ copy_xact_xlog_xid(void)
"pg_clog" : "pg_xact",
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ if (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
@@ -773,8 +775,8 @@ copy_xact_xlog_xid(void)
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ old_cluster.controldata.chkpnt_oldstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_oldstCommitTsxid,
+ old_cluster.controldata.chkpnt_newstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_newstCommitTsxid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 69c965bb7d0..b55e2c2ea84 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
--
2.42.2
Yes, the pg_commit_ts directory is not transferred by pg_upgrade.
Basically, you can get the exact same result by manually copying the
pg_commit_ts directory and running pg_resetwal. And this patch does it
automatically.
In other words, the patch provides the stated functionality, but
consider running pgindet.
I'm not sure whether there were any reasons not to make this move from
the beginning. The authors' opinions are needed here.
And one more thing. In my perspective, having some tests is always a
good idea, even if they are not meant to be committed. After briefly
examining the patch, I developed this basic test. This is not finalized
in any manner, but it is simply an idea.
--
Best regards,
Maxim Orlov.
Attachments:
Hi,
Per my understanding, track_commit_timestamp must be on the new instance,
otherwise it would be removed when new instance starts. Is it correct?
If so, should we detect the difference and do some warnings/errors here?
Or it is just the user's responsibility?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Dear Maxim,
Thanks for considering the test case. To confirm, where should we put the test code?
Pp_upgrade has already had 006 test, and commit_ts has not had 005 test.
Regarding the test, it is enough to check whether oldest/newest transactions on the old
cluster can be tracked.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
<div><div>Hi,</div><div> </div><div>Yes, track_commit_timestamp must be installed in the new instance.</div><div>This is only the responsibility of an experienced user.</div><div>pg_upgrage should allow you to save pg_commit_ts if this data exists at the time of migration.</div></div><div><div>Warnings are not needed, the loss of this data is not critical in most cases. They were lost with each migration if users did not manually migrate them.</div><div> </div></div><div>----------------</div><div>Кому: 'ls7777' (ls7777@yandex.ru);</div><div>Копия: pgsql-hackers@postgresql.org;</div><div>Тема: Patch for migration of the pg_commit_ts directory;</div><div>01.10.2025, 17:17, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>:</div><blockquote><p>Hi,<br /><br />Per my understanding, track_commit_timestamp must be on the new instance,<br />otherwise it would be removed when new instance starts. Is it correct?<br />If so, should we detect the difference and do some warnings/errors here?<br />Or it is just the user's responsibility?<br /><br />Best regards,<br />Hayato Kuroda<br />FUJITSU LIMITED<br /> </p></blockquote>
Hi,
(Sorry but I cannot find your name)
Yes, track_commit_timestamp must be installed in the new instance.
This is only the responsibility of an experienced user.
pg_upgrage should allow you to save pg_commit_ts if this data exists at the time of migration.
Warnings are not needed, the loss of this data is not critical in most cases.
They were lost with each migration if users did not manually migrate them.
So, your policy is that commit_ts is not the user data thus it is OK to drop during
the upgrade, is it correct? I want to know other's opinion around here.
Note that if we want to check, it can be done in check_control_data().
Regarding the patch:
```
-
+ cluster->controldata.chkpnt_oldstCommitTsxid = 0;
+ cluster->controldata.chkpnt_newstCommitTsxid = 0;
```
Other attributes are not initialized, you can follow.
```
+ if (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
```
Indent should be fixed. Please run pgindent.
```
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ old_cluster.controldata.chkpnt_oldstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_oldstCommitTsxid,
+ old_cluster.controldata.chkpnt_newstCommitTsxid == 0 ? old_cluster.controldata.chkpnt_nxtxid : old_cluster.controldata.chkpnt_newstCommitTsxid,
```
To confirm, is there a possibility that only either of oldest/newest CommitTs exists?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
On Thu, Oct 2, 2025 at 12:10 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
Yes, track_commit_timestamp must be installed in the new instance.
This is only the responsibility of an experienced user.
pg_upgrage should allow you to save pg_commit_ts if this data exists at the time of migration.
Warnings are not needed, the loss of this data is not critical in most cases.
They were lost with each migration if users did not manually migrate them.So, your policy is that commit_ts is not the user data thus it is OK to drop during
the upgrade, is it correct?
IIUC, the proposal is that if GUC track_commit_timestamp is enabled on
the new instance then we should copy it, otherwise, we can drop
copying it. Is my understanding correct? I think we can follow what is
done check_new_cluster_replication_slots() for the case when
track_commit_timestamp is not set on the new server. When we try to
copy slots and the wal_level on the new server is minimal, we error
out, so shouldn't we do the same here and error_out if
track_commit_timestamp is not enabled and we have some valid commit_ts
data to copy?
--
With Regards,
Amit Kapila.
On Thu, 2 Oct 2025 at 11:49, Amit Kapila <amit.kapila16@gmail.com> wrote:
When we try to
copy slots and the wal_level on the new server is minimal, we error
out, so shouldn't we do the same here and error_out if
track_commit_timestamp is not enabled and we have some valid commit_ts
data to copy?
+1 Sounds reasonable to me. It's better to give an explicit error;
otherwise, we would remove data that the user clearly wants to migrate
to the new cluster. And deleting data just because the user forgot to
specify one parameter in a new cluster looks like a bad joke to me.
--
Best regards,
Maxim Orlov.
<div><div>Hi,</div><div><div><span style="white-space:pre-wrap">I'll try to return it to the original </span><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">thread.</span></div><div> </div><div><div>I absolutely agree with you. Data loss during migration is an unacceptable situation. And why it's still happening, I don't understand. But issuing messages will require translating it into other languages and complicate the patch. I'm not a programmer to do it beautifully. And not a beautiful patch is unlikely to be accepted. </div><div>I will do a parameter check and an ERROR message.</div></div><div> </div></div><div> </div><blockquote><div><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Hi,</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Could you please reply to the original thread? Otherwise, it is difficult to track later.</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Regarding the patch, I think Amit suggested to raise an ERROR in case of parameter</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">mismatch, and Maxim agreed the point [2]. How do you feel?</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">[1]: </span><a href="/messages/by-id/CAA4eK1+Zayox=x84upet_gc4hrCKSwZSpnO9=Q4vH8rrJbsBOg@mail.gmail.com" rel="noopener noreferrer" target="_blank" style="background-color:rgb( 255 , 255 , 255 );color:rgb( 0 , 119 , 255 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">/messages/by-id/CAA4eK1+Zayox=x84upet_gc4hrCKSwZSpnO9=Q4vH8rrJbsBOg@mail.gmail.com</a><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">[2]: </span><a href="/messages/by-id/CACG=ezaFdL4EM8bdrHTRVHt5BmxxWGXcPiDr6Qm00Qj+S_Y+vQ@mail.gmail.com" rel="noopener noreferrer" target="_blank" style="background-color:rgb( 255 , 255 , 255 );color:rgb( 0 , 119 , 255 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">/messages/by-id/CACG=ezaFdL4EM8bdrHTRVHt5BmxxWGXcPiDr6Qm00Qj+S_Y+vQ@mail.gmail.com</a><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Best regards,</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Hayato Kuroda</span><br style="background-color:rgb( 255 , 255 , 255 );color:rgb( 26 , 26 , 26 );font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" /><span style="background-color:#ffffff;color:#1a1a1a;float:none;font-family:'ys text' , 'arial' , sans-serif;font-size:16px;font-style:normal;font-weight:400;text-align:start;text-decoration-color:initial;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">FUJITSU LIMITED</span></div></blockquote></div><div> </div><div>----------------</div><div>Кому: Amit Kapila (amit.kapila16@gmail.com);</div><div>Копия: Hayato Kuroda (Fujitsu) (kuroda.hayato@fujitsu.com), pgsql-hackers@postgresql.org;</div><div>Тема: Patch for migration of the pg_commit_ts directory;</div><div>06.10.2025, 12:59, "Maxim Orlov" <orlovmg@gmail.com>:</div><blockquote><div><div> </div> <div><div>On Thu, 2 Oct 2025 at 11:49, Amit Kapila <<a href="mailto:amit.kapila16@gmail.com" rel="noopener noreferrer">amit.kapila16@gmail.com</a>> wrote:</div><blockquote style="border-left-color:rgb( 204 , 204 , 204 );border-left-style:solid;border-left-width:1px;margin:0px 0px 0px 0.8ex;padding-left:1ex"> When we try to<br />copy slots and the wal_level on the new server is minimal, we error<br />out, so shouldn't we do the same here and error_out if<br />track_commit_timestamp is not enabled and we have some valid commit_ts<br />data to copy?</blockquote><div> </div><div>+1 Sounds reasonable to me. It's better to give an explicit error;<br />otherwise, we would remove data that the user clearly wants to migrate<br />to the new cluster. And deleting data just because the user forgot to<br />specify one parameter in a new cluster looks like a bad joke to me. </div><div> </div></div>-- <div><div><div>Best regards,</div><div>Maxim Orlov.</div></div></div></div></blockquote>
On Mon, Oct 6, 2025 at 07:31:52PM +0500, ls7777 wrote:
Hi,
I'll try to return it to the original thread.
I absolutely agree with you. Data loss during migration is an unacceptable
situation. And why it's still happening, I don't understand. But issuing
messages will require translating it into other languages and complicate the
patch. I'm not a programmer to do it beautifully. And not a beautiful patch is
unlikely to be accepted.
I will do a parameter check and an ERROR message.
You don't need to worry about translations. Each language team will
make sure your English message is translated.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Do not let urgent matters crowd out time for investment in the future.
Attachments:
v3-Migration-of-the-pg_commit_ts-directory.patchtext/x-diff; name=v3-Migration-of-the-pg_commit_ts-directory.patchDownload
From 64dbcbbb3b02eecfe9cc71deaaabe2140b199ab8 Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 16 +-
src/bin/pg_upgrade/controldata.c | 26 +++-
src/bin/pg_upgrade/info.c | 29 ++++
src/bin/pg_upgrade/pg_upgrade.c | 21 ++-
src/bin/pg_upgrade/pg_upgrade.h | 5 +
.../pg_upgrade/t/007_transfer_commit_ts.pl | 143 ++++++++++++++++++
6 files changed, 235 insertions(+), 5 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 1e17d64b3ec..b843d7d7e75 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -695,6 +696,8 @@ check_and_dump_old_cluster(void)
if (GET_MAJOR_VERSION(old_cluster.major_version) <= 905)
check_for_pg_role_prefix(&old_cluster);
+ check_track_commit_timestamp_parameter(&old_cluster);
+
/*
* While not a check option, we do this now because this is the only time
* the old server is running.
@@ -767,9 +770,20 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
-}
+ check_new_cluster_pg_commit_ts();
+
+}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ prep_status("Checking for pg_commit_ts");
+ check_track_commit_timestamp_parameter(&new_cluster);
+ if (old_cluster.track_commit_timestamp_on && !new_cluster.track_commit_timestamp_on)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 90cef0864de..b218ed92389 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -208,7 +208,11 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.data_checksum_version = 0;
got_data_checksum_version = true;
}
-
+ if (GET_MAJOR_VERSION(cluster->major_version) < 905)
+ {
+ cluster->controldata.chkpnt_oldstCommitTsxid = 0;
+ cluster->controldata.chkpnt_newstCommitTsxid = 0;
+ }
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -321,6 +325,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c
index 7ce08270168..d9b13b25a37 100644
--- a/src/bin/pg_upgrade/info.c
+++ b/src/bin/pg_upgrade/info.c
@@ -819,6 +819,35 @@ get_subscription_info(ClusterInfo *cluster)
PQfinish(conn);
}
+/*
+ * check_track_commit_timestamp_parameter(ClusterInfo *cluster)
+ *
+ * Gets track_commit_timestamp parameter in the cluster.
+ */
+void
+check_track_commit_timestamp_parameter(ClusterInfo *cluster)
+{
+ PGconn *conn;
+ PGresult *res;
+ int is_set;
+
+ /* There is a track_commit_timestamp parameter in 9.5 and above. */
+ if (GET_MAJOR_VERSION(cluster->major_version) < 905)
+ {
+ cluster->track_commit_timestamp_on = false;
+ }
+ else
+ {
+ conn = connectToServer(cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT count(*) AS is_set "
+ "FROM pg_settings WHERE name = 'track_commit_timestamp' and setting = 'on'");
+ is_set = PQfnumber(res, "is_set");
+ cluster->track_commit_timestamp_on = atoi(PQgetvalue(res, 0, is_set)) == 1;
+
+ PQclear(res);
+ PQfinish(conn);
+ }
+}
static void
free_db_and_rel_infos(DbInfoArr *db_arr)
{
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 490e98fa26f..dc91da78d9b 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -772,6 +772,8 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -781,6 +783,16 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy pg_commit_ts only if three conditions are met
+ */
+ is_copy_commit_ts = old_cluster.controldata.chkpnt_oldstCommitTsxid > 0
+ && old_cluster.controldata.chkpnt_newstCommitTsxid > 0
+ && new_cluster.track_commit_timestamp_on;
+
+ if (is_copy_commit_ts)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -798,12 +810,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_oldstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_newstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 0ef47be0dc1..4bbfdfa881e 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
@@ -306,6 +308,8 @@ typedef struct
int nsubs; /* number of subscriptions */
bool sub_retain_dead_tuples; /* whether a subscription enables
* retain_dead_tuples. */
+ bool track_commit_timestamp_on; /* track_commit_timestamp for
+ * cluster */
} ClusterInfo;
@@ -444,6 +448,7 @@ FileNameMap *gen_db_file_maps(DbInfo *old_db,
void get_db_rel_and_slot_infos(ClusterInfo *cluster);
int count_old_cluster_logical_slots(void);
void get_subscription_info(ClusterInfo *cluster);
+void check_track_commit_timestamp_parameter(ClusterInfo *cluster);
/* option.c */
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
new file mode 100644
index 00000000000..bbf0e2e1b29
--- /dev/null
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -0,0 +1,143 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+sub command_output
+{
+ my ($cmd) = @_;
+ my ($stdout, $stderr);
+
+ print("# Running: " . join(" ", @{$cmd}) . "\n");
+
+ my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+ ok($result, "@$cmd exit code 0");
+ is($stderr, '', "@$cmd no stderr");
+
+ return $stdout;
+}
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+$old->command_ok([ 'pgbench', '-i', '-s', '1' ], 'init pgbench');
+$old->command_ok([ 'pgbench', '-t', '100', '-j', '2' ], 'pgbench it');
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+command_fails(
+ [
+ 'pg_upgrade', '--no-sync',
+ '-d', $old->data_dir,
+ '-D', $new->data_dir,
+ '-b', $old->config_data('--bindir'),
+ '-B', $new->config_data('--bindir'),
+ '-s', $new->host,
+ '-p', $old->port,
+ '-P', $new->port,
+ $mode
+ ],
+ 'run of pg_upgrade fails with mismatch parameter track_commit_timestamp');
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+command_ok(
+ [
+ 'pg_upgrade', '--no-sync',
+ '-d', $old->data_dir,
+ '-D', $new->data_dir,
+ '-b', $old->config_data('--bindir'),
+ '-B', $new->config_data('--bindir'),
+ '-s', $new->host,
+ '-p', $old->port,
+ '-P', $new->port,
+ $mode
+ ],
+ 'run of pg_upgrade ok');
+
+
+$new->start;
+$new->command_ok([ 'pgbench', '-t', '10' ], 'pgbench it');
+$new->stop;
+
+sub xact_commit_ts
+{
+ my ($node) = @_;
+ my ($oldest, $newest);
+ my $out = command_output([ 'pg_controldata', '-D', $node->data_dir ]);
+
+ if ($out =~ /oldestCommitTsXid:(\d+)/) {
+ $oldest = $1;
+ }
+ if ($out =~ /newestCommitTsXid:(\d+)/) {
+ $newest= $1;
+ }
+
+ for my $line (grep { /\S/ } split /\n/, $out) {
+ if ($line =~ /Latest checkpoint's NextXID:/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's oldestXID:/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's oldestCommitTsXid/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's newestCommitTsXid:/) {
+ print $line . "\n";
+ }
+ }
+
+ $node->start;
+ my $res = $node->safe_psql('postgres', "
+ WITH xids AS (
+ SELECT v::text::xid AS x
+ FROM generate_series($oldest, $newest) v
+ )
+ SELECT x, pg_xact_commit_timestamp(x::text::xid)
+ FROM xids;
+ ");
+ $node->stop;
+ return grep { /\S/ } split /\n/, $res;
+}
+
+my @a = xact_commit_ts($old);
+my @b = xact_commit_ts($new);
+my %h1 = map { $_ => 1 } @a;
+my %h2 = map { $_ => 1 } @b;
+
+#
+# All timestamps from the old cluster should appear in the new one.
+#
+my $count = 0;
+print "Commit timestamp only in old cluster:\n";
+for my $line (@a) {
+ unless ($h2{$line}) {
+ print "$line\n";
+ $count = $count + 1;
+ }
+}
+ok($count == 0, "all the timestamp transferred successfully");
+
+#
+# The new cluster should contain timestamps created during the pg_upgrade and
+# some more created by the pgbench.
+#
+print "\nCommit timestamp only in new cluster:\n";
+for my $line (@b) {
+ print "$line\n" unless $h1{$line};
+}
+
+done_testing();
--
2.42.2
Hi,
Thanks for updating the patch.
Regarding the check_track_commit_timestamp_parameter(), I'm not sure the function
is needed. Since pg_controldata outputs whether the commit_ts module is enabled
or not [1]``` $ pg_controldata data/ | grep "track_" track_commit_timestamp setting: off ```, can we obtain there instead?
I imagined like we can add a new field at ControlData, it could be obtained in get_control_data(),
values for instances could be compared in check_control_data().
Another possibility is that oldestCommitTsXid/newestCommitTsXid can be always
valid if track_commit_timestamp is on. In this case the attribute is not needed
anymore. It is enough to check these values. Can you check them?
Regarding the test, I feel it is unnecessary long. I feel it is enough to ensure
one or two transactions have the same timestamp on the both nodes, pgbench is not
needed. Current test may not be able to finish on the slow machine.
[1]: ``` $ pg_controldata data/ | grep "track_" track_commit_timestamp setting: off ```
```
$ pg_controldata data/ | grep "track_"
track_commit_timestamp setting: off
```
Best regards,
Hayato Kuroda
FUJITSU LIMITED
<div>Hi,</div><div> </div><div><div>At the time check_control_data is launched for the new cluster, the control file does not contain information about the set track_commit_timestamp=on. It is installed only in postgresql.conf. The check_track_commit_timestamp_parameter function is only needed for the new cluster. And you can not run it for the old cluster, but use data from the old_cluster.controldata.chkpnt_newstCommitTsxid control file. But this will be less clear than using the check_track_commit_timestamp_parameter function.</div><div> </div><div>The test can be shortened.</div></div><div> </div><div>----------------</div><div>Кому: 'ls7777' (ls7777@yandex.ru), orlovmg@gmail.com (orlovmg@gmail.com), amit.kapila16@gmail.com (amit.kapila16@gmail.com);</div><div>Копия: pgsql-hackers@postgresql.org;</div><div>Тема: Patch for migration of the pg_commit_ts directory;</div><div>07.10.2025, 08:15, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>:</div><blockquote><p>Hi,<br /><br />Thanks for updating the patch.<br /><br />Regarding the check_track_commit_timestamp_parameter(), I'm not sure the function<br />is needed. Since pg_controldata outputs whether the commit_ts module is enabled<br />or not [1], can we obtain there instead?<br />I imagined like we can add a new field at ControlData, it could be obtained in get_control_data(),<br />values for instances could be compared in check_control_data().<br /><br />Another possibility is that oldestCommitTsXid/newestCommitTsXid can be always<br />valid if track_commit_timestamp is on. In this case the attribute is not needed<br />anymore. It is enough to check these values. Can you check them?<br /><br />Regarding the test, I feel it is unnecessary long. I feel it is enough to ensure<br />one or two transactions have the same timestamp on the both nodes, pgbench is not<br />needed. Current test may not be able to finish on the slow machine.<br /><br />[1]<br />```<br />$ pg_controldata data/ | grep "track_"<br />track_commit_timestamp setting: off<br />```<br /><br />Best regards,<br />Hayato Kuroda<br />FUJITSU LIMITED<br /> </p></blockquote>
Hi,
At the time check_control_data is launched for the new cluster, the control file
does not contain information about the set track_commit_timestamp=on. It is
installed only in postgresql.conf.
You did consider the case that track_commit_timestamp is set to on but the instance
has not started before the pg_upgrade, right? Valid point.
The check_track_commit_timestamp_parameter function is only needed for the new
cluster. And you can not run it for the old cluster, but use data from the
old_cluster.controldata.chkpnt_newstCommitTsxid control file. But this will be
less clear than using the check_track_commit_timestamp_parameter function.
Sorry, I'm not very clear this part. Can you describe the point bit more?
Other comments:
01. get_control_data
```
+ if (GET_MAJOR_VERSION(cluster->major_version) < 905)
+ {
+ cluster->controldata.chkpnt_oldstCommitTsxid = 0;
+ cluster->controldata.chkpnt_newstCommitTsxid = 0;
+ }
```
IIUC, checksum_version is checked like this because got_data_checksum_version
must be also set.
Since we do not have the boolean for commit_ts, not sure it is needed.
old_cluster and new_cluster are the global variable and they are initialized with
zero.
02. check_track_commit_timestamp_parameter
```
+ conn = connectToServer(cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT count(*) AS is_set "
+ "FROM pg_settings WHERE name = 'track_commit_timestamp' and setting = 'on'");
+ is_set = PQfnumber(res, "is_set");
+ cluster->track_commit_timestamp_on = atoi(PQgetvalue(res, 0, is_set)) == 1;
```
The SQL looks strange for me. Isn't it enough to directly obtain "setting" attribute
and check it is "on" or not? Also, conn and res can be the local variable of the
else part.
03. ClusterInfo
```
+ bool track_commit_timestamp_on; /* track_commit_timestamp for
+ * cluster */
```
The indent is broken, please run pgindent.
04. test
Could you please update meson.build? Otherwise the added test could not be run for meson build.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v4-Migration-of-the-pg_commit_ts-directory.patchtext/x-diff; name=v4-Migration-of-the-pg_commit_ts-directory.patchDownload
From 1d5499d87687ca15dd7d4daf1b98d68dd11dba74 Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 22 +++
src/bin/pg_upgrade/controldata.c | 21 ++-
src/bin/pg_upgrade/meson.build | 1 +
src/bin/pg_upgrade/pg_upgrade.c | 20 ++-
src/bin/pg_upgrade/pg_upgrade.h | 2 +
.../pg_upgrade/t/007_transfer_commit_ts.pl | 139 ++++++++++++++++++
6 files changed, 201 insertions(+), 4 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 1e17d64b3ec..f0e25dd3191 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -767,9 +768,30 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
+
+ check_new_cluster_pg_commit_ts();
+
}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ bool track_commit_timestamp_on;
+
+ prep_status("Checking for pg_commit_ts");
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT setting "
+ "FROM pg_settings WHERE name = 'track_commit_timestamp'");
+ track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
+ PQclear(res);
+ PQfinish(conn);
+ if (old_cluster.controldata.chkpnt_newstCommitTsxid > 0 && !track_commit_timestamp_on)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 90cef0864de..991000265ad 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -208,7 +208,6 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.data_checksum_version = 0;
got_data_checksum_version = true;
}
-
/* we have the result of cmd in "output". so parse it line by line now */
while (fgets(bufin, sizeof(bufin), output))
{
@@ -321,6 +320,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/meson.build b/src/bin/pg_upgrade/meson.build
index ac992f0d14b..030618152d0 100644
--- a/src/bin/pg_upgrade/meson.build
+++ b/src/bin/pg_upgrade/meson.build
@@ -47,6 +47,7 @@ tests += {
't/004_subscription.pl',
't/005_char_signedness.pl',
't/006_transfer_modes.pl',
+ 't/007_transfer_commit_ts.pl',
],
'test_kwargs': {'priority': 40}, # pg_upgrade tests are slow
},
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 490e98fa26f..98d01e8600c 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -772,6 +772,8 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -781,6 +783,15 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy pg_commit_ts
+ */
+ is_copy_commit_ts = old_cluster.controldata.chkpnt_oldstCommitTsxid > 0
+ && old_cluster.controldata.chkpnt_newstCommitTsxid > 0;
+
+ if (is_copy_commit_ts)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -798,12 +809,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_oldstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_newstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 0ef47be0dc1..83fdbc44b13 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
new file mode 100644
index 00000000000..97d0b88ff2a
--- /dev/null
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+sub command_output
+{
+ my ($cmd) = @_;
+ my ($stdout, $stderr);
+
+ print("# Running: " . join(" ", @{$cmd}) . "\n");
+
+ my $result = IPC::Run::run $cmd, '>', \$stdout, '2>', \$stderr;
+ ok($result, "@$cmd exit code 0");
+ is($stderr, '', "@$cmd no stderr");
+
+ return $stdout;
+}
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+$old->command_ok([ 'pgbench', '-i', '-s', '1' ], 'init pgbench');
+$old->command_ok([ 'pgbench', '-t', '1', '-j', '1' ], 'pgbench it');
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+command_fails(
+ [
+ 'pg_upgrade', '--no-sync',
+ '-d', $old->data_dir,
+ '-D', $new->data_dir,
+ '-b', $old->config_data('--bindir'),
+ '-B', $new->config_data('--bindir'),
+ '-s', $new->host,
+ '-p', $old->port,
+ '-P', $new->port,
+ $mode
+ ],
+ 'run of pg_upgrade fails with mismatch parameter track_commit_timestamp');
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+command_ok(
+ [
+ 'pg_upgrade', '--no-sync',
+ '-d', $old->data_dir,
+ '-D', $new->data_dir,
+ '-b', $old->config_data('--bindir'),
+ '-B', $new->config_data('--bindir'),
+ '-s', $new->host,
+ '-p', $old->port,
+ '-P', $new->port,
+ $mode
+ ],
+ 'run of pg_upgrade ok');
+
+
+sub xact_commit_ts
+{
+ my ($node) = @_;
+ my ($oldest, $newest);
+ my $out = command_output([ 'pg_controldata', '-D', $node->data_dir ]);
+
+ if ($out =~ /oldestCommitTsXid:(\d+)/) {
+ $oldest = $1;
+ }
+ if ($out =~ /newestCommitTsXid:(\d+)/) {
+ $newest= $1;
+ }
+
+ for my $line (grep { /\S/ } split /\n/, $out) {
+ if ($line =~ /Latest checkpoint's NextXID:/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's oldestXID:/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's oldestCommitTsXid/) {
+ print $line . "\n";
+ }
+ if ($line =~ /Latest checkpoint's newestCommitTsXid:/) {
+ print $line . "\n";
+ }
+ }
+
+ $node->start;
+ my $res = $node->safe_psql('postgres', "
+ WITH xids AS (
+ SELECT v::text::xid AS x
+ FROM generate_series($oldest, $newest) v
+ )
+ SELECT x, pg_xact_commit_timestamp(x::text::xid)
+ FROM xids;
+ ");
+ $node->stop;
+ return grep { /\S/ } split /\n/, $res;
+}
+
+my @a = xact_commit_ts($old);
+my @b = xact_commit_ts($new);
+my %h1 = map { $_ => 1 } @a;
+my %h2 = map { $_ => 1 } @b;
+
+#
+# All timestamps from the old cluster should appear in the new one.
+#
+my $count = 0;
+print "Commit timestamp only in old cluster:\n";
+for my $line (@a) {
+ unless ($h2{$line}) {
+ print "$line\n";
+ $count = $count + 1;
+ }
+}
+ok($count == 0, "all the timestamp transferred successfully");
+
+#
+# The new cluster should contain timestamps created during the pg_upgrade and
+# some more created by the pgbench.
+#
+print "\nCommit timestamp only in new cluster:\n";
+for my $line (@b) {
+ print "$line\n" unless $h1{$line};
+}
+
+done_testing();
--
2.42.2
Hi,
Thanks for updating the patch and sorry for the late reply. Here are my comments.
01.
```
+ prep_status("Checking for pg_commit_ts");
```
I think we must clarify which node is being checked. Something like:
Checking for new cluster configuration for commit timestamp
02.
```
}
-
/* we have the result of cmd in "output". so parse it line by line now */
```
This change is not needed.
03.
```
+ /*
+ * Copy pg_commit_ts
+ */
```
I feel it can be removed or have more meanings. Something lile:
Copy old commit_timestamp data to new, if available.
04.
Regarding the test,
05.
```
+sub command_output
```
Can run_command() be used instead of defining new function?
06.
```
+$old->command_ok([ 'pgbench', '-i', '-s', '1' ], 'init pgbench');
+$old->command_ok([ 'pgbench', '-t', '1', '-j', '1' ], 'pgbench it');
```
I think no need to use pgbench anymore. Can we define tables and insert tuples
by myself?
07.
```
+command_fails(
+ [
+ 'pg_upgrade', '--no-sync',
+ '-d', $old->data_dir,
+ '-D', $new->data_dir,
+ '-b', $old->config_data('--bindir'),
+ '-B', $new->config_data('--bindir'),
+ '-s', $new->host,
+ '-p', $old->port,
+ '-P', $new->port,
+ $mode
+ ],
+ 'run of pg_upgrade fails with mismatch parameter track_commit_timestamp');
```
According to other test files, we do not use the shorten option.
Also, please verify the actual output by command_ok_or_fails_like() or command_checks_all().
08.
```
+sub xact_commit_ts
```
Not sure, but this function is introduced because we have 100 transactions. Can we omit this now?
09.
```
+# The new cluster should contain timestamps created during the pg_upgrade and
+# some more created by the pgbench.
+#
+print "\nCommit timestamp only in new cluster:\n";
+for my $line (@b) {
+ print "$line\n" unless $h1{$line};
+}
```
I don't think this is needed because the output is not verified.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v5-Migration-of-the-pg_commit_ts-directory.patchtext/x-diff; name=v5-Migration-of-the-pg_commit_ts-directory.patchDownload
From 254cae551bbde68c4932d61dc96167fee9f97155 Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 22 ++++++
src/bin/pg_upgrade/controldata.c | 20 +++++
src/bin/pg_upgrade/meson.build | 1 +
src/bin/pg_upgrade/pg_upgrade.c | 20 ++++-
src/bin/pg_upgrade/pg_upgrade.h | 2 +
.../pg_upgrade/t/007_transfer_commit_ts.pl | 75 +++++++++++++++++++
6 files changed, 137 insertions(+), 3 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 1e17d64b3ec..c5e29485416 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -767,9 +768,30 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
+
+ check_new_cluster_pg_commit_ts();
+
}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ bool track_commit_timestamp_on;
+
+ prep_status("Checking for new cluster configuration for commit timestamp");
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT setting "
+ "FROM pg_settings WHERE name = 'track_commit_timestamp'");
+ track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
+ PQclear(res);
+ PQfinish(conn);
+ if (old_cluster.controldata.chkpnt_newstCommitTsxid > 0 && !track_commit_timestamp_on)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 90cef0864de..594dc1c0a9f 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -321,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/meson.build b/src/bin/pg_upgrade/meson.build
index ac992f0d14b..030618152d0 100644
--- a/src/bin/pg_upgrade/meson.build
+++ b/src/bin/pg_upgrade/meson.build
@@ -47,6 +47,7 @@ tests += {
't/004_subscription.pl',
't/005_char_signedness.pl',
't/006_transfer_modes.pl',
+ 't/007_transfer_commit_ts.pl',
],
'test_kwargs': {'priority': 40}, # pg_upgrade tests are slow
},
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 490e98fa26f..c46a4432fd2 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -772,6 +772,8 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -781,6 +783,15 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy old commit_timestamp data to new, if available.
+ */
+ is_copy_commit_ts = old_cluster.controldata.chkpnt_oldstCommitTsxid > 0
+ && old_cluster.controldata.chkpnt_newstCommitTsxid > 0;
+
+ if (is_copy_commit_ts)
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -798,12 +809,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_oldstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
+ is_copy_commit_ts ? old_cluster.controldata.chkpnt_newstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 0ef47be0dc1..83fdbc44b13 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
new file mode 100644
index 00000000000..1b75bf17e8c
--- /dev/null
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -0,0 +1,75 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+my $resold = $old->safe_psql('postgres', "
+ create table a(a int);
+ select xid,timestamp from pg_last_committed_xact();
+ ");
+
+my ($xid,$ts) = $resold =~ /\s*(\d+)\s*\|(.*)/;
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+command_checks_all(
+ [
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode
+ ],
+ 1,
+ [qr{"track_commit_timestamp" must be "on" but is set to "off"}],
+ [],
+ 'run of pg_upgrade for mismatch parameter track_commit_timestamp');
+
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+
+command_checks_all(
+ [
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode
+ ],
+ 0,
+ [],
+ [],
+ 'run of pg_upgrade ok');
+
+$new->start;
+my $resnew = $new->safe_psql('postgres', "
+ select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
+ ");
+$new->stop;
+ok($resold eq $resnew, "timestamp transferred successfully");
+
+done_testing();
--
2.42.2
Looks good to me. As I see it, the patch is almost ready, so I decided
to create an CF entry [0]https://commitfest.postgresql.org/patch/6119/. Feel free to edit it as you see fit.
And your mail also includes attachment (unknown_filename
</messages/by-id/attachment/175614/unknown_filename>
), interfering with the CF-bot.
<div><div>Thanks.</div><div>That's the good news.</div><div><div><div><span style="white-space:pre-wrap">Thanks to Hayato Kuroda, for the advice.</span></div><div><span style="white-space:pre-wrap">Thanks to everyone who participated in the promotion of the patch.</span></div><div> </div></div></div></div><div> </div><div>----------------</div><div>Кому: ls7777 (ls7777@yandex.ru);</div><div>Копия: Hayato Kuroda (Fujitsu) (kuroda.hayato@fujitsu.com), pgsql-hackers@postgresql.org, amit.kapila16@gmail.com;</div><div>Тема: Patch for migration of the pg_commit_ts directory;</div><div>10.10.2025, 15:55, "Maxim Orlov" <orlovmg@gmail.com>:</div><blockquote><div><div><span lang="en">Looks good to me. As I see it, the patch is almost ready, so I decided </span></div><div><span lang="en">to create an CF entry [0]. Feel free to edit it as you see fit.</span></div><div> </div><div><span lang="en">And your mail also includes a</span>ttachment (<a href="/messages/by-id/attachment/175614/unknown_filename" rel="noopener noreferrer">unknown_filename</a>), <span lang="en">interfering with the CF-bot.</span></div><div> </div><div>[0] <a href="https://commitfest.postgresql.org/patch/6119/" rel="noopener noreferrer">https://commitfest.postgresql.org/patch/6119/</a></div></div></blockquote>
Hi,
Thanks for updating the patch.
```
+my ($xid,$ts) = $resold =~ /\s*(\d+)\s*\|(.*)/;
```
Per my understanding $ts is not used here. Can you remove it?
Apart from above, I found some cosmetic issues. Please see attached
my fix which can be applied atop HEAD. Can you check and include if
it is OK?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
kuroda.diffsapplication/octet-stream; name=kuroda.diffsDownload
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index c5e29485416..382da0485a7 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -782,14 +782,16 @@ check_new_cluster_pg_commit_ts(void)
prep_status("Checking for new cluster configuration for commit timestamp");
conn = connectToServer(&new_cluster, "template1");
- res = executeQueryOrDie(conn, "SELECT setting "
- "FROM pg_settings WHERE name = 'track_commit_timestamp'");
+ res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'track_commit_timestamp'");
track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
PQclear(res);
PQfinish(conn);
- if (old_cluster.controldata.chkpnt_newstCommitTsxid > 0 && !track_commit_timestamp_on)
+ if (!track_commit_timestamp_on &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0)
pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+
check_ok();
}
void
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index c46a4432fd2..70297f38dca 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -773,6 +773,7 @@ static void
copy_xact_xlog_xid(void)
{
bool is_copy_commit_ts;
+ uint32 oldest_xid, newest_xid;
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
@@ -786,11 +787,18 @@ copy_xact_xlog_xid(void)
/*
* Copy old commit_timestamp data to new, if available.
*/
- is_copy_commit_ts = old_cluster.controldata.chkpnt_oldstCommitTsxid > 0
- && old_cluster.controldata.chkpnt_newstCommitTsxid > 0;
+ is_copy_commit_ts =
+ (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0 &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0);
if (is_copy_commit_ts)
+ {
copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+ oldest_xid = old_cluster.controldata.chkpnt_oldstCommitTsxid;
+ newest_xid = old_cluster.controldata.chkpnt_newstCommitTsxid;
+ }
+ else
+ oldest_xid = newest_xid = old_cluster.controldata.chkpnt_nxtxid;
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
@@ -816,8 +824,8 @@ copy_xact_xlog_xid(void)
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- is_copy_commit_ts ? old_cluster.controldata.chkpnt_oldstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
- is_copy_commit_ts ? old_cluster.controldata.chkpnt_newstCommitTsxid : old_cluster.controldata.chkpnt_nxtxid,
+ oldest_xid,
+ newest_xid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
index 1b75bf17e8c..c67463ab961 100644
--- a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -17,58 +17,50 @@ my $old = PostgreSQL::Test::Cluster->new('old');
$old->init;
$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$old->start;
-my $resold = $old->safe_psql('postgres', "
+my $resold = $old->safe_psql(
+ 'postgres', qq{
create table a(a int);
- select xid,timestamp from pg_last_committed_xact();
- ");
+ select xid,timestamp from pg_last_committed_xact();
+});
-my ($xid,$ts) = $resold =~ /\s*(\d+)\s*\|(.*)/;
+my ($xid, $ts) = $resold =~ /\s*(\d+)\s*\|(.*)/;
$old->stop;
# Initialize new cluster
my $new = PostgreSQL::Test::Cluster->new('new');
$new->init;
+
+# Setup a common pg_upgrade command to be used by all the test cases
+my @pg_upgrade_cmd = (
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode);
+
+# In a VPATH build, we'll be started in the source directory, but we want
+# to run pg_upgrade in the build directory so that any files generated finish
+# in it, like delete_old_cluster.{sh,bat}.
+chdir ${PostgreSQL::Test::Utils::tmp_check};
+
command_checks_all(
- [
- 'pg_upgrade', '--no-sync',
- '--old-datadir' => $old->data_dir,
- '--new-datadir' => $new->data_dir,
- '--old-bindir' => $old->config_data('--bindir'),
- '--new-bindir' => $new->config_data('--bindir'),
- '--socketdir' => $new->host,
- '--old-port' => $old->port,
- '--new-port' => $new->port,
- $mode
- ],
- 1,
- [qr{"track_commit_timestamp" must be "on" but is set to "off"}],
- [],
+ [@pg_upgrade_cmd], 1,
+ [qr{"track_commit_timestamp" must be "on" but is set to "off"}], [],
'run of pg_upgrade for mismatch parameter track_commit_timestamp');
-
$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
-command_checks_all(
- [
- 'pg_upgrade', '--no-sync',
- '--old-datadir' => $old->data_dir,
- '--new-datadir' => $new->data_dir,
- '--old-bindir' => $old->config_data('--bindir'),
- '--new-bindir' => $new->config_data('--bindir'),
- '--socketdir' => $new->host,
- '--old-port' => $old->port,
- '--new-port' => $new->port,
- $mode
- ],
- 0,
- [],
- [],
- 'run of pg_upgrade ok');
+command_ok([@pg_upgrade_cmd], 'run of pg_upgrade ok');
$new->start;
-my $resnew = $new->safe_psql('postgres', "
- select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
- ");
+my $resnew = $new->safe_psql(
+ 'postgres', qq{
+ select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
+});
$new->stop;
ok($resold eq $resnew, "timestamp transferred successfully");
Attachments:
v6-Migration-of-the-pg_commit_ts-directory.patchtext/x-diff; name=v6-Migration-of-the-pg_commit_ts-directory.patchDownload
From bc582227cdd51d252db00fccc4d572524f9a4b91 Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 24 +++++++
src/bin/pg_upgrade/controldata.c | 20 ++++++
src/bin/pg_upgrade/meson.build | 1 +
src/bin/pg_upgrade/pg_upgrade.c | 28 +++++++-
src/bin/pg_upgrade/pg_upgrade.h | 2 +
.../pg_upgrade/t/007_transfer_commit_ts.pl | 67 +++++++++++++++++++
6 files changed, 139 insertions(+), 3 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 1e17d64b3ec..382da0485a7 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -767,9 +768,32 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
+
+ check_new_cluster_pg_commit_ts();
+
}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ bool track_commit_timestamp_on;
+
+ prep_status("Checking for new cluster configuration for commit timestamp");
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'track_commit_timestamp'");
+ track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
+ PQclear(res);
+ PQfinish(conn);
+ if (!track_commit_timestamp_on &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 90cef0864de..594dc1c0a9f 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -321,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/meson.build b/src/bin/pg_upgrade/meson.build
index ac992f0d14b..030618152d0 100644
--- a/src/bin/pg_upgrade/meson.build
+++ b/src/bin/pg_upgrade/meson.build
@@ -47,6 +47,7 @@ tests += {
't/004_subscription.pl',
't/005_char_signedness.pl',
't/006_transfer_modes.pl',
+ 't/007_transfer_commit_ts.pl',
],
'test_kwargs': {'priority': 40}, # pg_upgrade tests are slow
},
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 490e98fa26f..70297f38dca 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -772,6 +772,9 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+ uint32 oldest_xid, newest_xid;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -781,6 +784,22 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy old commit_timestamp data to new, if available.
+ */
+ is_copy_commit_ts =
+ (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0 &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0);
+
+ if (is_copy_commit_ts)
+ {
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+ oldest_xid = old_cluster.controldata.chkpnt_oldstCommitTsxid;
+ newest_xid = old_cluster.controldata.chkpnt_newstCommitTsxid;
+ }
+ else
+ oldest_xid = newest_xid = old_cluster.controldata.chkpnt_nxtxid;
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -798,12 +817,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ oldest_xid,
+ newest_xid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index 0ef47be0dc1..83fdbc44b13 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
new file mode 100644
index 00000000000..e4ae1d951fa
--- /dev/null
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -0,0 +1,67 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+my $resold = $old->safe_psql(
+ 'postgres', qq{
+ create table a(a int);
+ select xid,timestamp from pg_last_committed_xact();
+});
+
+my ($xid) = $resold =~ /\s*(\d+)\s*\|.*/;
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+
+# Setup a common pg_upgrade command to be used by all the test cases
+my @pg_upgrade_cmd = (
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode);
+
+# In a VPATH build, we'll be started in the source directory, but we want
+# to run pg_upgrade in the build directory so that any files generated finish
+# in it, like delete_old_cluster.{sh,bat}.
+chdir ${PostgreSQL::Test::Utils::tmp_check};
+
+command_checks_all(
+ [@pg_upgrade_cmd], 1,
+ [qr{"track_commit_timestamp" must be "on" but is set to "off"}], [],
+ 'run of pg_upgrade for mismatch parameter track_commit_timestamp');
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+
+command_ok([@pg_upgrade_cmd], 'run of pg_upgrade ok');
+
+$new->start;
+my $resnew = $new->safe_psql(
+ 'postgres', qq{
+ select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
+});
+$new->stop;
+ok($resold eq $resnew, "timestamp transferred successfully");
+
+done_testing();
--
2.42.2
Hi,
Thanks for updating the patch. Mostly looks good.
I ran pgindent locally and indents of my part were incorrect. PSA the
included version.
I have no comments anymore. Thanks for the great work.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v7-0001-Migration-of-the-pg_commit_ts-directory.patchapplication/octet-stream; name=v7-0001-Migration-of-the-pg_commit_ts-directory.patchDownload
From b8f529a6ed706e5064f3962d77c0d3128534947c Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH v7] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 24 +++++++
src/bin/pg_upgrade/controldata.c | 20 ++++++
src/bin/pg_upgrade/meson.build | 1 +
src/bin/pg_upgrade/pg_upgrade.c | 29 +++++++-
src/bin/pg_upgrade/pg_upgrade.h | 2 +
.../pg_upgrade/t/007_transfer_commit_ts.pl | 67 +++++++++++++++++++
6 files changed, 140 insertions(+), 3 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index 1e17d64b3ec..73c69236d4d 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -767,9 +768,32 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
+
+ check_new_cluster_pg_commit_ts();
+
}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ bool track_commit_timestamp_on;
+
+ prep_status("Checking for new cluster configuration for commit timestamp");
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'track_commit_timestamp'");
+ track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
+ PQclear(res);
+ PQfinish(conn);
+ if (!track_commit_timestamp_on &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index 90cef0864de..594dc1c0a9f 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -321,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/meson.build b/src/bin/pg_upgrade/meson.build
index ac992f0d14b..030618152d0 100644
--- a/src/bin/pg_upgrade/meson.build
+++ b/src/bin/pg_upgrade/meson.build
@@ -47,6 +47,7 @@ tests += {
't/004_subscription.pl',
't/005_char_signedness.pl',
't/006_transfer_modes.pl',
+ 't/007_transfer_commit_ts.pl',
],
'test_kwargs': {'priority': 40}, # pg_upgrade tests are slow
},
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 490e98fa26f..3566a2a75bc 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -772,6 +772,10 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+ uint32 oldest_xid,
+ newest_xid;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -781,6 +785,22 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy old commit_timestamp data to new, if available.
+ */
+ is_copy_commit_ts =
+ (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0 &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0);
+
+ if (is_copy_commit_ts)
+ {
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+ oldest_xid = old_cluster.controldata.chkpnt_oldstCommitTsxid;
+ newest_xid = old_cluster.controldata.chkpnt_newstCommitTsxid;
+ }
+ else
+ oldest_xid = newest_xid = old_cluster.controldata.chkpnt_nxtxid;
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -798,12 +818,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ oldest_xid,
+ newest_xid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index e86336f4be9..ccc391311a3 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -238,6 +238,8 @@ typedef struct
uint32 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
diff --git a/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
new file mode 100644
index 00000000000..e4ae1d951fa
--- /dev/null
+++ b/src/bin/pg_upgrade/t/007_transfer_commit_ts.pl
@@ -0,0 +1,67 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+my $resold = $old->safe_psql(
+ 'postgres', qq{
+ create table a(a int);
+ select xid,timestamp from pg_last_committed_xact();
+});
+
+my ($xid) = $resold =~ /\s*(\d+)\s*\|.*/;
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+
+# Setup a common pg_upgrade command to be used by all the test cases
+my @pg_upgrade_cmd = (
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode);
+
+# In a VPATH build, we'll be started in the source directory, but we want
+# to run pg_upgrade in the build directory so that any files generated finish
+# in it, like delete_old_cluster.{sh,bat}.
+chdir ${PostgreSQL::Test::Utils::tmp_check};
+
+command_checks_all(
+ [@pg_upgrade_cmd], 1,
+ [qr{"track_commit_timestamp" must be "on" but is set to "off"}], [],
+ 'run of pg_upgrade for mismatch parameter track_commit_timestamp');
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+
+command_ok([@pg_upgrade_cmd], 'run of pg_upgrade ok');
+
+$new->start;
+my $resnew = $new->safe_psql(
+ 'postgres', qq{
+ select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
+});
+$new->stop;
+ok($resold eq $resnew, "timestamp transferred successfully");
+
+done_testing();
--
2.47.3
On Wed, 15 Oct 2025 at 04:46, Hayato Kuroda (Fujitsu) <
kuroda.hayato@fujitsu.com> wrote:
Hi,
Thanks for updating the patch. Mostly looks good.
Yep, looks good to me too.
If there are no objections, I will move forward to mark the thread as
ready for committers.
Hayato Kuroda, you did a great job reviewing this patch; consider to
tag yourself as a reviewer [0]https://commitfest.postgresql.org/patch/6119/.
[0]: https://commitfest.postgresql.org/patch/6119/
--
Best regards,
Maxim Orlov.
Dear Hackers,
I found that v7 needs rebased. Copyright was also updated in the attached patch.
I'm not the author of the patch though.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
v8-0001-Migration-of-the-pg_commit_ts-directory.patchapplication/octet-stream; name=v8-0001-Migration-of-the-pg_commit_ts-directory.patchDownload
From 53cde45863d0fa00c6dc84a3dd02ab10a622b6bd Mon Sep 17 00:00:00 2001
From: Sergey Levin <ls7777@yandex.ru>
Date: Sat, 4 Oct 2025 20:22:46 +0500
Subject: [PATCH v8] Migration of the pg_commit_ts directory
---
src/bin/pg_upgrade/check.c | 24 +++++++
src/bin/pg_upgrade/controldata.c | 20 ++++++
src/bin/pg_upgrade/meson.build | 1 +
src/bin/pg_upgrade/pg_upgrade.c | 29 +++++++-
src/bin/pg_upgrade/pg_upgrade.h | 2 +
.../pg_upgrade/t/008_transfer_commit_ts.pl | 67 +++++++++++++++++++
6 files changed, 140 insertions(+), 3 deletions(-)
create mode 100644 src/bin/pg_upgrade/t/008_transfer_commit_ts.pl
diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c
index a47d553b809..fc0b720b60c 100644
--- a/src/bin/pg_upgrade/check.c
+++ b/src/bin/pg_upgrade/check.c
@@ -32,6 +32,7 @@ static void check_new_cluster_replication_slots(void);
static void check_new_cluster_subscription_configuration(void);
static void check_old_cluster_for_valid_slots(void);
static void check_old_cluster_subscription_state(void);
+static void check_new_cluster_pg_commit_ts(void);
/*
* DataTypesUsageChecks - definitions of data type checks for the old cluster
@@ -767,9 +768,32 @@ check_new_cluster(void)
check_new_cluster_replication_slots();
check_new_cluster_subscription_configuration();
+
+ check_new_cluster_pg_commit_ts();
+
}
+void
+check_new_cluster_pg_commit_ts(void)
+{
+ PGconn *conn;
+ PGresult *res;
+ bool track_commit_timestamp_on;
+
+ prep_status("Checking for new cluster configuration for commit timestamp");
+ conn = connectToServer(&new_cluster, "template1");
+ res = executeQueryOrDie(conn, "SELECT setting FROM pg_settings "
+ "WHERE name = 'track_commit_timestamp'");
+ track_commit_timestamp_on = strcmp(PQgetvalue(res, 0, 0), "on") == 0;
+ PQclear(res);
+ PQfinish(conn);
+ if (!track_commit_timestamp_on &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0)
+ pg_fatal("\"track_commit_timestamp\" must be \"on\" but is set to \"off\"");
+
+ check_ok();
+}
void
report_clusters_compatible(void)
{
diff --git a/src/bin/pg_upgrade/controldata.c b/src/bin/pg_upgrade/controldata.c
index aa6e8b4de5d..fa8b28adf43 100644
--- a/src/bin/pg_upgrade/controldata.c
+++ b/src/bin/pg_upgrade/controldata.c
@@ -321,6 +321,26 @@ get_control_data(ClusterInfo *cluster)
cluster->controldata.chkpnt_nxtmulti = str2uint(p);
got_multi = true;
}
+ else if ((p = strstr(bufin, "Latest checkpoint's oldestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_oldstCommitTsxid = str2uint(p);
+ }
+ else if ((p = strstr(bufin, "Latest checkpoint's newestCommitTsXid:")) != NULL)
+ {
+ p = strchr(p, ':');
+
+ if (p == NULL || strlen(p) <= 1)
+ pg_fatal("%d: controldata retrieval problem", __LINE__);
+
+ p++; /* remove ':' char */
+ cluster->controldata.chkpnt_newstCommitTsxid = str2uint(p);
+ }
else if ((p = strstr(bufin, "Latest checkpoint's oldestXID:")) != NULL)
{
p = strchr(p, ':');
diff --git a/src/bin/pg_upgrade/meson.build b/src/bin/pg_upgrade/meson.build
index 49b1b624f25..b27477c3f8a 100644
--- a/src/bin/pg_upgrade/meson.build
+++ b/src/bin/pg_upgrade/meson.build
@@ -51,6 +51,7 @@ tests += {
't/005_char_signedness.pl',
't/006_transfer_modes.pl',
't/007_multixact_conversion.pl',
+ 't/008_transfer_commit_ts.pl',
],
'test_kwargs': {'priority': 40}, # pg_upgrade tests are slow
},
diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c
index 2127d297bfe..b761f6c86c4 100644
--- a/src/bin/pg_upgrade/pg_upgrade.c
+++ b/src/bin/pg_upgrade/pg_upgrade.c
@@ -773,6 +773,10 @@ copy_subdir_files(const char *old_subdir, const char *new_subdir)
static void
copy_xact_xlog_xid(void)
{
+ bool is_copy_commit_ts;
+ uint32 oldest_xid,
+ newest_xid;
+
/*
* Copy old commit logs to new data dir. pg_clog has been renamed to
* pg_xact in post-10 clusters.
@@ -782,6 +786,22 @@ copy_xact_xlog_xid(void)
GET_MAJOR_VERSION(new_cluster.major_version) <= 906 ?
"pg_clog" : "pg_xact");
+ /*
+ * Copy old commit_timestamp data to new, if available.
+ */
+ is_copy_commit_ts =
+ (old_cluster.controldata.chkpnt_oldstCommitTsxid > 0 &&
+ old_cluster.controldata.chkpnt_newstCommitTsxid > 0);
+
+ if (is_copy_commit_ts)
+ {
+ copy_subdir_files("pg_commit_ts", "pg_commit_ts");
+ oldest_xid = old_cluster.controldata.chkpnt_oldstCommitTsxid;
+ newest_xid = old_cluster.controldata.chkpnt_newstCommitTsxid;
+ }
+ else
+ oldest_xid = newest_xid = old_cluster.controldata.chkpnt_nxtxid;
+
prep_status("Setting oldest XID for new cluster");
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -u %u \"%s\"",
@@ -799,12 +819,15 @@ copy_xact_xlog_xid(void)
"\"%s/pg_resetwal\" -f -e %u \"%s\"",
new_cluster.bindir, old_cluster.controldata.chkpnt_nxtepoch,
new_cluster.pgdata);
- /* must reset commit timestamp limits also */
+
+ /*
+ * must reset commit timestamp limits also or copy from the old cluster
+ */
exec_prog(UTILITY_LOG_FILE, NULL, true, true,
"\"%s/pg_resetwal\" -f -c %u,%u \"%s\"",
new_cluster.bindir,
- old_cluster.controldata.chkpnt_nxtxid,
- old_cluster.controldata.chkpnt_nxtxid,
+ oldest_xid,
+ newest_xid,
new_cluster.pgdata);
check_ok();
diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h
index ec018e4f292..938413521cd 100644
--- a/src/bin/pg_upgrade/pg_upgrade.h
+++ b/src/bin/pg_upgrade/pg_upgrade.h
@@ -245,6 +245,8 @@ typedef struct
uint64 chkpnt_nxtmxoff;
uint32 chkpnt_oldstMulti;
uint32 chkpnt_oldstxid;
+ uint32 chkpnt_oldstCommitTsxid;
+ uint32 chkpnt_newstCommitTsxid;
uint32 align;
uint32 blocksz;
uint32 largesz;
diff --git a/src/bin/pg_upgrade/t/008_transfer_commit_ts.pl b/src/bin/pg_upgrade/t/008_transfer_commit_ts.pl
new file mode 100644
index 00000000000..a9f0ff4ba8f
--- /dev/null
+++ b/src/bin/pg_upgrade/t/008_transfer_commit_ts.pl
@@ -0,0 +1,67 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Tests for transfer pg_commit_ts directory.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Can be changed to test the other modes
+my $mode = $ENV{PG_TEST_PG_UPGRADE_MODE} || '--copy';
+
+# Initialize old cluster
+my $old = PostgreSQL::Test::Cluster->new('old');
+$old->init;
+$old->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$old->start;
+my $resold = $old->safe_psql(
+ 'postgres', qq{
+ create table a(a int);
+ select xid,timestamp from pg_last_committed_xact();
+});
+
+my ($xid) = $resold =~ /\s*(\d+)\s*\|.*/;
+$old->stop;
+
+# Initialize new cluster
+my $new = PostgreSQL::Test::Cluster->new('new');
+$new->init;
+
+# Setup a common pg_upgrade command to be used by all the test cases
+my @pg_upgrade_cmd = (
+ 'pg_upgrade', '--no-sync',
+ '--old-datadir' => $old->data_dir,
+ '--new-datadir' => $new->data_dir,
+ '--old-bindir' => $old->config_data('--bindir'),
+ '--new-bindir' => $new->config_data('--bindir'),
+ '--socketdir' => $new->host,
+ '--old-port' => $old->port,
+ '--new-port' => $new->port,
+ $mode);
+
+# In a VPATH build, we'll be started in the source directory, but we want
+# to run pg_upgrade in the build directory so that any files generated finish
+# in it, like delete_old_cluster.{sh,bat}.
+chdir ${PostgreSQL::Test::Utils::tmp_check};
+
+command_checks_all(
+ [@pg_upgrade_cmd], 1,
+ [qr{"track_commit_timestamp" must be "on" but is set to "off"}], [],
+ 'run of pg_upgrade for mismatch parameter track_commit_timestamp');
+
+$new->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+
+command_ok([@pg_upgrade_cmd], 'run of pg_upgrade ok');
+
+$new->start;
+my $resnew = $new->safe_psql(
+ 'postgres', qq{
+ select $xid,pg_xact_commit_timestamp(${xid}::text::xid);
+});
+$new->stop;
+ok($resold eq $resnew, "timestamp transferred successfully");
+
+done_testing();
--
2.47.3
<div><div>Hi,</div><div> </div><div><div><div>I didn't quite understand what I needed to do. </div><div>My assumptions:</div><div>1. You need to download the postgresql master branch and create a patch file on it.</div><div> </div><div>2. Replace the 2025 with 2026 header comment in all с<span style="white-space:pre-wrap">hangeable</span> patch files.</div><div> </div><div>Did I understand correctly?</div></div></div></div><div> </div><div>----------------</div><div>Кому: 'ls7777' (ls7777@yandex.ru);</div><div>Копия: pgsql-hackers@postgresql.org, orlovmg@gmail.com, amit.kapila16@gmail.com;</div><div>Тема: Patch for migration of the pg_commit_ts directory;</div><div>05.01.2026, 13:25, "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>:</div><blockquote><p>Dear Hackers,<br /><br />I found that v7 needs rebased. Copyright was also updated in the attached patch.<br />I'm not the author of the patch though.<br /><br />Best regards,<br />Hayato Kuroda<br />FUJITSU LIMITED<br /> </p></blockquote>
Hi,
I didn't quite understand what I needed to do.
My assumptions:
1. You need to download the postgresql master branch and create a patch file on it.
This is correct. I periodically checked the commitfest app [1]https://commitfest.postgresql.org/patch/6119/, and it has
reported [Needs rebase!] for weeks. Now it could be applied cleanly and tests
could be passed.
2. Replace the 2025 with 2026 header comment in all сhangeable patch files.
Since I cannot follow the sentence, let me clarify my understanding.
Basically, the copyright notation is maintained by the PostgreSQL community;
nothing for us to do. They are updated at the beginning of the year [2]https://github.com/postgres/postgres/commit/451c43974f8e199097d97624a4952ad0973cea61.
If you are proposing to add new files, however, they must contain the copyright
and be updated in the new year. It's not yet included in the codebase and is out
of scope for the community's maintenance.
Please ask me anything if you have more questions :-).
[1]: https://commitfest.postgresql.org/patch/6119/
[2]: https://github.com/postgres/postgres/commit/451c43974f8e199097d97624a4952ad0973cea61
Best regards,
Hayato Kuroda
FUJITSU LIMITED