Add 64-bit XIDs into PostgreSQL 15

Hi Maxim,

I’m glad to see that you’re trying to carry the 64-bit XID work forward. I had not noticed that my earlier patch (also derived from Alexander Kortkov’s patch) was responded to back in September. Perhaps we can merge some of the code cleanup that it contained, such as using XID_FMT everywhere and creating a type for the kind of page returned by TransactionIdToPage() to make the code cleaner.

Is your patch functionally the same as the PostgresPro implementation? If so, I think it would be helpful for everyone’s understanding to read the PostgresPro documentation on VACUUM. See in particular section “Forced shrinking pg_clog and pg_multixact”

https://postgrespro.com/docs/enterprise/9.6/routine-vacuuming#vacuum-for-wraparound

best regards,

/Jim

Greetings,

* Maxim Orlov (orlovmg@gmail.com) wrote:

Long time wraparound was a really big pain for highly loaded systems. One
source of performance degradation is the need to vacuum before every
wraparound.
And there were several proposals to make XIDs 64-bit like [1], [2], [3] and
[4] to name a few.

The approach [2] seems to have stalled on CF since 2018. But meanwhile it
was successfully being used in our Postgres Pro fork all time since then.
We have hundreds of customers using 64-bit XIDs. Dozens of instances are
under load that require wraparound each 1-5 days with 32-bit XIDs.
It really helps the customers with a huge transaction load that in the case
of 32-bit XIDs could experience wraparounds every day. So I'd like to
propose this approach modification to CF.

PFA updated working patch v6 for PG15 development cycle.
It is based on a patch by Alexander Korotkov version 5 [5] with a few
fixes, refactoring and was rebased to PG15.

Just to confirm as I only did a quick look- if a transaction in such a
high rate system lasts for more than a day (which certainly isn't
completely out of the question, I've had week-long transactions
before..), and something tries to delete a tuple which has tuples on it
that can't be frozen yet due to the long-running transaction- it's just
going to fail?

Not saying that I've got any idea how to fix that case offhand, and we
don't really support such a thing today as the server would just stop
instead, but if I saw something in the release notes talking about PG
moving to 64bit transaction IDs, I'd probably be pretty surprised to
discover that there's still a 32bit limit that you have to watch out for
or your system will just start failing transactions. Perhaps that's a
worthwhile tradeoff for being able to generally avoid having to vacuum
and deal with transaction wrap-around, but I have to wonder if there
might be a better answer. Of course, also wonder about how we're going
to document and monitor for this potential issue and what kind of
corrective action will be needed (kill transactions older than a cerain
amount of transactions..?).

Thanks,

Stephen

On 1/4/22, 2:35 PM, "Stephen Frost" <sfrost@snowman.net> wrote:

Not saying that I've got any idea how to fix that case offhand, and we
don't really support such a thing today as the server would just stop
instead, ...
Perhaps that's a
worthwhile tradeoff for being able to generally avoid having to vacuum
and deal with transaction wrap-around, but I have to wonder if there
might be a better answer.

For the target use cases that PostgreSQL is designed for, it's a very worthwhile tradeoff in my opinion. Such long-running transactions need to be killed.

Re: -- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.

I'm concerned about the maintainability impact of having 2 new on-disk page formats. It's already complex enough with XIDs and multixact-XIDs.

If the lack of space for the two epochs in the special data area is a problem only in an upgrade scenario, why not resolve the problem before completing the upgrade process like a kind of post-process pg_repack operation that converts all "double xmax" pages to the "double-epoch" page format? i.e. maybe the "double xmax" representation is needed as an intermediate representation during upgrade, but after upgrade completes successfully there are no pages with the "double-xmax" representation. This would eliminate a whole class of coding errors and would make the code dealing with 64-bit XIDs simpler and more maintainable.

/Jim

On 2021/12/30 21:15, Maxim Orlov wrote:

Hi, hackers!

Long time wraparound was a really big pain for highly loaded systems. One source of performance degradation is the need to vacuum before every wraparound.
And there were several proposals to make XIDs 64-bit like [1], [2], [3] and [4] to name a few.

The approach [2] seems to have stalled on CF since 2018. But meanwhile it was successfully being used in our Postgres Pro fork all time since then. We have hundreds of customers using 64-bit XIDs. Dozens of instances are under load that require wraparound each 1-5 days with 32-bit XIDs.
It really helps the customers with a huge transaction load that in the case of 32-bit XIDs could experience wraparounds every day. So I'd like to propose this approach modification to CF.

PFA updated working patch v6 for PG15 development cycle.
It is based on a patch by Alexander Korotkov version 5 [5] with a few fixes, refactoring and was rebased to PG15.

Thanks a lot! I'm really happy to see this proposal again!!

Is there any documentation or README explaining this whole 64-bit XID mechanism?

Could you tell me what happens if new tuple with XID larger than xid_base + 0xFFFFFFFF is inserted into the page? Such new tuple is not allowed to be inserted into that page? Or xid_base and xids of all existing tuples in the page are increased? Also what happens if one of those xids (of existing tuples) cannot be changed because the tuple still can be seen by very-long-running transaction?

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On Thu, 30 Dec 2021 at 13:19, Maxim Orlov <orlovmg@gmail.com> wrote:

Hi, hackers!

Long time wraparound was a really big pain for highly loaded systems. One source of performance degradation is the need to vacuum before every wraparound.
And there were several proposals to make XIDs 64-bit like [1], [2], [3] and [4] to name a few.

Very good to see this revived.

PFA updated working patch v6 for PG15 development cycle.
It is based on a patch by Alexander Korotkov version 5 [5] with a few fixes, refactoring and was rebased to PG15.

Main changes:
- Change TransactionId to 64bit

This sounds like a good route to me.

- Disk tuple format (HeapTupleHeader) is unchanged: xmin and xmax
remains 32bit
-- 32bit xid is named ShortTransactionId now.
-- Exception: see "double xmax" format below.
- Heap page format is changed to contain xid and multixact base value,
tuple's xmin and xmax are offsets from.
-- xid_base and multi_base are stored as a page special data. PageHeader
remains unmodified.
-- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.
- In-memory tuples (HeapTuple) were enriched with copies of xid_base and
multi_base from a page.

I think we need more Overview of Behavior than is available with this
patch, perhaps in the form of a README, such as in
src/backend/access/heap/README.HOT.

Most people's comments are about what the opportunities and problems
caused, and mine are definitely there also. i.e. explain the user
visible behavior.
Please explain the various new states that pages can be in and what
the effects are,

My understanding is this would be backwards compatible, so we can
upgrade to it. Please confirm.

Thanks

--
Simon Riggs http://www.EnterpriseDB.com/

On Tue, Jan 4, 2022 at 10:22:50PM +0000, Finnerty, Jim wrote:

I'm concerned about the maintainability impact of having 2 new
on-disk page formats. It's already complex enough with XIDs and
multixact-XIDs.

If the lack of space for the two epochs in the special data area is
a problem only in an upgrade scenario, why not resolve the problem
before completing the upgrade process like a kind of post-process
pg_repack operation that converts all "double xmax" pages to
the "double-epoch" page format? i.e. maybe the "double xmax"
representation is needed as an intermediate representation during
upgrade, but after upgrade completes successfully there are no pages
with the "double-xmax" representation. This would eliminate a whole
class of coding errors and would make the code dealing with 64-bit
XIDs simpler and more maintainable.

Well, yes, we could do this, and it would avoid the complexity of having
to support two XID representations, but we would need to accept that
fast pg_upgrade would be impossible in such cases, since every page
would need to be checked and potentially updated.

You might try to do this while the server is first started and running
queries, but I think we found out from the online checkpoint patch that
having the server in an intermediate state while running queries is very
complex --- it might be simpler to just accept two XID formats all the
time than enabling the server to run with two formats for a short
period. My big point is that this needs more thought.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

On Thu, Dec 30, 2021 at 03:15:16PM +0300, Maxim Orlov wrote:

PFA updated working patch v6 for PG15 development cycle.
It is based on a patch by Alexander Korotkov version 5 [5] with a few fixes,
refactoring and was rebased to PG15.

Main changes:
- Change TransactionId to 64bit
- Disk tuple format (HeapTupleHeader) is unchanged: xmin and xmax
  remains 32bit
-- 32bit xid is named ShortTransactionId now.
-- Exception: see "double xmax" format below.
- Heap page format is changed to contain xid and multixact base value,
  tuple's xmin and xmax are offsets from.
-- xid_base and multi_base are stored as a page special data. PageHeader
   remains unmodified.
-- If after upgrade page has no free space for special data, tuples are
   converted to "double xmax" format: xmin became virtual
   FrozenTransactionId, xmax occupies the whole 64bit.
   Page converted to new format when vacuum frees enough space.

I think it is a great idea to allow the 64-XID to span the 32-bit xmin
and xmax fields when needed. It would be nice if we can get focus on
this feature so we are sure it gets into PG 15. Can we add this patch
incrementally so people can more easily analyze it?

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

On Tue, Jan 4, 2022 at 05:49:07PM +0000, Finnerty, Jim wrote:

Hi Maxim,
I’m glad to see that you’re trying to carry the 64-bit XID work forward. I
had not noticed that my earlier patch (also derived from Alexander Kortkov’s
patch) was responded to back in September. Perhaps we can merge some of the
code cleanup that it contained, such as using XID_FMT everywhere and creating a
type for the kind of page returned by TransactionIdToPage() to make the code
cleaner.

Is your patch functionally the same as the PostgresPro implementation? If
so, I think it would be helpful for everyone’s understanding to read the
PostgresPro documentation on VACUUM. See in particular section “Forced
shrinking pg_clog and pg_multixact”

https://postgrespro.com/docs/enterprise/9.6/routine-vacuuming#
vacuum-for-wraparound

Good point --- we still need vacuum freeze. It would be good to
understand how much value we get in allowing vacuum freeze to be done
less often --- how big can pg_xact/pg_multixact get before they are
problems?

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

On Wed, Jan 05, 2022 at 06:51:37PM -0500, Bruce Momjian wrote:

On Tue, Jan 4, 2022 at 10:22:50PM +0000, Finnerty, Jim wrote:

I'm concerned about the maintainability impact of having 2 new
on-disk page formats. It's already complex enough with XIDs and
multixact-XIDs.

If the lack of space for the two epochs in the special data area is
a problem only in an upgrade scenario, why not resolve the problem
before completing the upgrade process like a kind of post-process
pg_repack operation that converts all "double xmax" pages to
the "double-epoch" page format? i.e. maybe the "double xmax"
representation is needed as an intermediate representation during
upgrade, but after upgrade completes successfully there are no pages
with the "double-xmax" representation. This would eliminate a whole
class of coding errors and would make the code dealing with 64-bit
XIDs simpler and more maintainable.

Well, yes, we could do this, and it would avoid the complexity of having
to support two XID representations, but we would need to accept that
fast pg_upgrade would be impossible in such cases, since every page
would need to be checked and potentially updated.

You might try to do this while the server is first started and running
queries, but I think we found out from the online checkpoint patch that

I think you meant the online checksum patch. Which this reminded me of, too.

https://commitfest.postgresql.org/31/2611/

having the server in an intermediate state while running queries is very
complex --- it might be simpler to just accept two XID formats all the
time than enabling the server to run with two formats for a short
period. My big point is that this needs more thought.

--
Justin

On Wed, Jan 5, 2022 at 06:12:26PM -0600, Justin Pryzby wrote:

On Wed, Jan 05, 2022 at 06:51:37PM -0500, Bruce Momjian wrote:

On Tue, Jan 4, 2022 at 10:22:50PM +0000, Finnerty, Jim wrote:

I'm concerned about the maintainability impact of having 2 new
on-disk page formats. It's already complex enough with XIDs and
multixact-XIDs.

If the lack of space for the two epochs in the special data area is
a problem only in an upgrade scenario, why not resolve the problem
before completing the upgrade process like a kind of post-process
pg_repack operation that converts all "double xmax" pages to
the "double-epoch" page format? i.e. maybe the "double xmax"
representation is needed as an intermediate representation during
upgrade, but after upgrade completes successfully there are no pages
with the "double-xmax" representation. This would eliminate a whole
class of coding errors and would make the code dealing with 64-bit
XIDs simpler and more maintainable.

Well, yes, we could do this, and it would avoid the complexity of having
to support two XID representations, but we would need to accept that
fast pg_upgrade would be impossible in such cases, since every page
would need to be checked and potentially updated.

You might try to do this while the server is first started and running
queries, but I think we found out from the online checkpoint patch that

I think you meant the online checksum patch. Which this reminded me of, too.

https://commitfest.postgresql.org/31/2611/

Sorry, yes, I have checkpoint on my mind. ;-)

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

Hi!

On Thu, Jan 6, 2022 at 3:02 AM Bruce Momjian <bruce@momjian.us> wrote:

On Thu, Dec 30, 2021 at 03:15:16PM +0300, Maxim Orlov wrote:

PFA updated working patch v6 for PG15 development cycle.
It is based on a patch by Alexander Korotkov version 5 [5] with a few fixes,
refactoring and was rebased to PG15.

Main changes:
- Change TransactionId to 64bit
- Disk tuple format (HeapTupleHeader) is unchanged: xmin and xmax
remains 32bit
-- 32bit xid is named ShortTransactionId now.
-- Exception: see "double xmax" format below.
- Heap page format is changed to contain xid and multixact base value,
tuple's xmin and xmax are offsets from.
-- xid_base and multi_base are stored as a page special data. PageHeader
remains unmodified.
-- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.

I think it is a great idea to allow the 64-XID to span the 32-bit xmin
and xmax fields when needed. It would be nice if we can get focus on
this feature so we are sure it gets into PG 15. Can we add this patch
incrementally so people can more easily analyze it?

I see at least the following major issues/questions in this patch.
1) Current code relies on the fact that TransactionId can be
atomically read from/written to shared memory. With 32-bit systems
and 64-bit TransactionId, that's not true anymore. Therefore, the
patch has concurrency issues on 32-bit systems. We need to carefully
review these issues and provide a fallback for 32-bit systems. I
suppose nobody is thinking about dropping off 32-bit systems, right?
Also, I wonder how critical for us is the overhead for 32-bit systems.
They are going to become legacy, so overhead isn't so critical, right?
2) With this patch we still need to freeze to cut SLRUs. This is
especially problematic with Multixacts, because systems heavily using
row-level locks can consume an enormous amount of multixacts. That is
especially problematic when we have 2x bigger multixacts. We probably
can provide an alternative implementation for multixact vacuum, which
doesn't require scanning all the heaps. That is a pretty amount of
work though. The clog is much smaller and we can cut it more rarely.
Perhaps, we could tolerate freezing to cut clog, couldn't we?
3) 2x bigger in-memory representation of TransactionId have overheads.
In particular, it could mitigate the effect of recent advancements
from Andres Freund. I'm not exactly sure we should/can do something
with this. But I think this should be at least carefully analyzed.
4) SP-GiST index stores TransactionId on pages. Could we tolerate
dropping SP-GiST indexes on a major upgrade? Or do we have to invent
something smarter?
5) 32-bit limitation within the page mentioned upthread by Stephen
Frost should be also carefully considered. Ideally, we should get rid
of it, but I don't have particular ideas in this field for now. At
least, we should make sure we did our best at error reporting and
monitoring capabilities.

I think the realistic goal for PG 15 development cycle would be
agreement on a roadmap for all the items above (and probably some
initial implementations).

------
Regards,
Alexander Korotkov

On Thu, 30 Dec 2021 at 13:19, Maxim Orlov <orlovmg@gmail.com> wrote:

Your opinions are very much welcome!

This is a review of the Int64 options patch,
"v6-0001-Add-64-bit-GUCs-for-xids.patch"

Applies cleanly, with some fuzz, compiles cleanly and passes make check.
Patch eyeballs OK, no obvious defects.
Tested using the attached test, so seems to work correctly.
On review of docs, no additions or changes required.
Perhaps add something to README? If so, minor doc patch attached.

Otherwise, this sub-patch is READY FOR COMMITTER.

--
Simon Riggs http://www.EnterpriseDB.com/

(Maxim) Re: -- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.

A better way would be to prepare the database for conversion to the 64-bit XID format before the upgrade so that it ensures that every page has enough room for the two new epochs (E bits).

1. Enforce the rule that no INSERT or UPDATE to an existing page will leave less than E bits of free space on a heap page

2. Run an online and restartable task, analogous to pg_repack, that rewrites and splits any page that has less than E bits of free space. This needs to be run on all non-temp tables in all schemas in all databases. DDL operations are not allowed on a target table while this operation runs, which is enforced by taking an ACCESS SHARE lock on each table while the process is running. To mitigate the effects of this restriction, the restartable task can be restricted to run only in certain hours. This could be implemented as a background maintenance task that runs for X hours as of a certain time of day and then kicks itself off again in 24-X hours, logging its progress.

When this task completes, the database is ready for upgrade to 64-bit XIDs, and there is no possibility that any page has insufficient free space for the special data.

Would you agree that this approach would completely eliminate the need for a "double xmax" representation?

/Jim

On Thu, 6 Jan 2022 at 13:15, Finnerty, Jim <jfinnert@amazon.com> wrote:

(Maxim) Re: -- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.

A better way would be to prepare the database for conversion to the 64-bit XID format before the upgrade so that it ensures that every page has enough room for the two new epochs (E bits).

Most software has a one-stage upgrade model. What you propose would
have us install 2 things, with a step in-between, which makes it
harder to manage.

1. Enforce the rule that no INSERT or UPDATE to an existing page will leave less than E bits of free space on a heap page

2. Run an online and restartable task, analogous to pg_repack, that rewrites and splits any page that has less than E bits of free space. This needs to be run on all non-temp tables in all schemas in all databases. DDL operations are not allowed on a target table while this operation runs, which is enforced by taking an ACCESS SHARE lock on each table while the process is running. To mitigate the effects of this restriction, the restartable task can be restricted to run only in certain hours. This could be implemented as a background maintenance task that runs for X hours as of a certain time of day and then kicks itself off again in 24-X hours, logging its progress.

When this task completes, the database is ready for upgrade to 64-bit XIDs, and there is no possibility that any page has insufficient free space for the special data.

Would you agree that this approach would completely eliminate the need for a "double xmax" representation?

I agree about the idea for scanning existing data blocks, but why not
do this AFTER upgrade?

1. Upgrade, with important aspect not-enabled-yet, but everything else
working - all required software is delivered in one shot, with fast
upgrade
2. As each table is VACUUMed, we confirm/clean/groom data blocks so
each table is individually confirmed as being ready. The pace that
this happens at is under user control.
3. When all tables have been prepared, then restart to allow xid64 format usage

--
Simon Riggs http://www.EnterpriseDB.com/

Re: Most software has a one-stage upgrade model. What you propose would
have us install 2 things, with a step in-between, which makes it
harder to manage.

The intended benefit would be that the code doesn't need to handle the possibility of 2 different XID representations for the indefinite future.

I agree that VACUUM would be the preferred tool to make room for the special data area so that there is no need to install a separate tool, though, whether this work happens before or after the upgrade.

Re: 1. Upgrade, with important aspect not-enabled-yet, but everything else working - all required software is delivered in one shot, with fast upgrade

Let's clarify what happens during upgrade. What format are the pages in immediately after the upgrade?

2. As each table is VACUUMed, we confirm/clean/groom data blocks so
each table is individually confirmed as being ready. The pace that
this happens at is under user control.

What are VACUUM's new responsibilities in this phase? VACUUM needs a new task that confirms when there exists no heap page for a table that is not ready.

If upgrade put all the pages into either double-xmax or double-epoch representation, then VACUUM's responsibility could be to split the double-xmax pages into the double-epoch representation and verify when there exists no double-xmax pages.

3. When all tables have been prepared, then restart to allow xid64 format usage

Let's also clarify what happens at restart time.

If we were to do the upgrade before preparing space in advance, is there a way to ever remove the code that knows about the double-xmax XID representation?

On Tue, Jan 4, 2022 at 9:40 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:

Could you tell me what happens if new tuple with XID larger than xid_base + 0xFFFFFFFF is inserted into the page? Such new tuple is not allowed to be inserted into that page?

I fear that this patch will have many bugs along these lines. Example:
Why is it okay that convert_page() may have to defragment a heap page,
without holding a cleanup lock? That will subtly break code that holds
a pin on the buffer, when a tuple slot contains a C pointer to a
HeapTuple in shared memory (though only if we get unlucky).

Currently we are very permissive about what XID/backend can (or
cannot) consume a specific piece of free space from a specific heap
page in code like RelationGetBufferForTuple(). It would be very hard
to enforce a policy like "your XID cannot insert onto this particular
heap page" with the current FSM design. (I actually think that we
should have a FSM that supports these requirements, but that's a big
project -- a long term goal of mine [1]/messages/by-id/CAH2-Wz=zEV4y_wxh-A_EvKxeAoCMdquYMHABEh_kZO1rk3a-gw@mail.gmail.com -- Peter Geoghegan.)

Or xid_base and xids of all existing tuples in the page are increased? Also what happens if one of those xids (of existing tuples) cannot be changed because the tuple still can be seen by very-long-running transaction?

That doesn't work in the general case, I think. How could it, unless
we truly had 64-bit XIDs in heap tuple headers? You can't necessarily
freeze to fix the problem, because we can only freeze XIDs that 1.)
committed, and 2.) are visible to every possible MVCC snapshot. (I
think you were alluding to this problem yourself.)

I believe that a good solution to the problem that this patch tries to
solve needs to be more ambitious. I think that we need to return to
first principles, rather than extending what we have already.
Currently, we store XIDs in tuple headers so that we can determine the
tuple's visibility status, based on whether the XID committed (or
aborted), and where our snapshot sees the XID as "in the past" (in the
case of an inserted tuple's xmin). You could say that XIDs from tuple
headers exist so we can see differences *between* tuples. But these
differences are typically not useful/interesting for very long. 32-bit
XIDs are sometimes not wide enough, but usually they're "too wide":
Why should we need to consider an old XID (e.g. do clog lookups) at
all, barring extreme cases?

Why do we need to keep any kind of metadata about transactions around
for a long time? Postgres has not supported time travel in 25 years!

If we eagerly cleaned-up aborted transactions with a special kind of
VACUUM (which would remove aborted XIDs), we could also maintain a
structure that indicates if all of the XIDs on a heap page are known
"all committed" implicitly (no dirtying the page, no hint bits, etc)
-- something a little like the visibility map, that is mostly set
implicitly (not during VACUUM). That doesn't fix the wraparound
problem itself, of course. But it enables a design that imposes the
same problem on the specific old snapshot instead -- something like a
"snapshot too old" error is much better than a system-wide wraparound
failure. That approach is definitely very hard, and also requires a
smart FSM along the lines described in [1]/messages/by-id/CAH2-Wz=zEV4y_wxh-A_EvKxeAoCMdquYMHABEh_kZO1rk3a-gw@mail.gmail.com -- Peter Geoghegan, but it seems like the best
way forward.

As I pointed out already, freezing is bad because it imposes the
requirement that everybody considers an affected XID committed and
visible, which is brittle (e.g., old snapshots can cause wraparound
failure). More generally, we rely too much on explicitly maintaining
"absolute" metadata inline, when we should implicitly maintain
"relative" metadata (that can be discarded quickly and without concern
for old snapshots). We need to be more disciplined about what XIDs can
modify what heap pages in the first place (in code like hio.c and the
FSM) to make all this work.

[1]: /messages/by-id/CAH2-Wz=zEV4y_wxh-A_EvKxeAoCMdquYMHABEh_kZO1rk3a-gw@mail.gmail.com -- Peter Geoghegan
--
Peter Geoghegan

,On Thu, Jan 6, 2022 at 4:15 PM Finnerty, Jim <jfinnert@amazon.com> wrote:

(Maxim) Re: -- If after upgrade page has no free space for special data, tuples are
converted to "double xmax" format: xmin became virtual
FrozenTransactionId, xmax occupies the whole 64bit.
Page converted to new format when vacuum frees enough space.

A better way would be to prepare the database for conversion to the 64-bit XID format before the upgrade so that it ensures that every page has enough room for the two new epochs (E bits).

1. Enforce the rule that no INSERT or UPDATE to an existing page will leave less than E bits of free space on a heap page

2. Run an online and restartable task, analogous to pg_repack, that rewrites and splits any page that has less than E bits of free space. This needs to be run on all non-temp tables in all schemas in all databases. DDL operations are not allowed on a target table while this operation runs, which is enforced by taking an ACCESS SHARE lock on each table while the process is running. To mitigate the effects of this restriction, the restartable task can be restricted to run only in certain hours. This could be implemented as a background maintenance task that runs for X hours as of a certain time of day and then kicks itself off again in 24-X hours, logging its progress.

When this task completes, the database is ready for upgrade to 64-bit XIDs, and there is no possibility that any page has insufficient free space for the special data.

Would you agree that this approach would completely eliminate the need for a "double xmax" representation?

The "prepare" approach was the first tried.
https://github.com/postgrespro/pg_pageprep
But it appears to be very difficult and unreliable. After investing
many months into pg_pageprep, "double xmax" approach appears to be
very fast to implement and reliable.

------
Regards,
Alexander Korotkov

On 2022/01/06 19:24, Simon Riggs wrote:

On Thu, 30 Dec 2021 at 13:19, Maxim Orlov <orlovmg@gmail.com> wrote:

Your opinions are very much welcome!

This is a review of the Int64 options patch,
"v6-0001-Add-64-bit-GUCs-for-xids.patch"

Do we really need to support both int32 and int64 options? Isn't it enough to replace the existing int32 option with int64 one? Or how about using string-type option for very large number like 64-bit XID, like it's done for recovery_target_xid?

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

Re: clog page numbers, as returned by TransactionIdToPage

-	int			pageno = TransactionIdToPage(xid);	/* get page of parent */
+	int64		pageno = TransactionIdToPage(xid);	/* get page of parent */

...

-	int			pageno = TransactionIdToPage(subxids[0]);
+	int64		pageno = TransactionIdToPage(subxids[0]);
 	int			offset = 0;
 	int			i = 0;

...

-		int			nextpageno;
+		int64		nextpageno;

Etc.

In all those places where you are replacing int with int64 for the kind of values returned by TransactionIdToPage(), would you mind replacing the int64's with a type name, such as ClogPageNumber, for improved code maintainability?