Next step towards 64bit XIDs: Switch to FullTransactionId for PGPROC->xid and XLogRecord->xl_xid

On Fri, 29 Dec 2023, 13:49 Maxim Orlov, <orlovmg@gmail.com> wrote:

Hi!

As were discussed in [0] our overall goal is to make Postgres 64 bit
XIDs. It's obvious, that such a big patch set
couldn't possible to commit "at once". SLUR patch set [1] was committed a
short while ago as a first significant
step in this direction.

This thread is a next step in this enterprise. My objective here is to
commit some changes, which were mandatory,
as far as I understand, for any type of 64 XIDs implementation. And I'm
sure there will be points for discussion here.

My original intention was to make PGPROC->xmin, PGPROC->xid and
PROC_HDR->xids 64bit. But in reality,
it turned out to be much more difficult than I expected. On the one hand,
the patch became too big and on the other
hand, it's heavily relayed on epoch and XID "adjustment" to FXID. Therefore,
for now, I decided to limit myself to
more atomic and independent changes. However, as I said above, these
changes are required for any implementation
of 64bit XIDs.

So, PFA patches to make switch PGPROC->xid

I think this could be fine, but ...

and XLogRecord->xl_xid to FullTransactionId.

I don't think this is an actionable change, as this wastes 4 more bytes (or
8 with alignment) in nearly all WAL records that don't use the
HEAP/HEAP2/XLOG rmgrs, which would then be up to 10 (if not 14, when
64but-aligned) bytes per record. Unless something like [0]https://commitfest.postgresql.org/46/4386/ gets committed
this will add a significant write overhead to all operations, even if they
are not doing anything that needs an XID.

Also, I don't think we need to support transactions that stay alive and
change things for longer than 2^31 concurrently created transactions, so we
could well add a fxid to each WAL segment header (and checkpoint record?)
and calculate the fxid of each record as a relative fxid off of that.

Kind regards

Matthias van de Meent

[0]: https://commitfest.postgresql.org/46/4386/

On Fri, Dec 29, 2023 at 02:36:19PM +0100, Matthias van de Meent wrote:

I don't think this is an actionable change, as this wastes 4 more bytes (or
8 with alignment) in nearly all WAL records that don't use the
HEAP/HEAP2/XLOG rmgrs, which would then be up to 10 (if not 14, when
64but-aligned) bytes per record. Unless something like [0] gets committed
this will add a significant write overhead to all operations, even if they
are not doing anything that needs an XID.

I was eyeing at the patches before reading your comment, and saw this
across the two patches:

@@ -176,7 +176,7 @@ struct PGPROC
Latch procLatch; /* generic latch for process */

-	TransactionId xid;			/* id of top-level transaction currently being
+	FullTransactionId xid;		/* id of top-level transaction currently being
[...]
 typedef struct XLogRecord
 {
 	uint32		xl_tot_len;		/* total len of entire record */
-	TransactionId xl_xid;		/* xact id */
+	pg_crc32c	xl_crc;			/* CRC for this record */
+	FullTransactionId xl_xid;	/* xact id */

And FWIW, echoing with Matthias, making these generic structures
arbitrary larger is a non-starter. We should try to make them
shorter.
--
Michael

On Fri, 29 Dec 2023 at 16:36, Matthias van de Meent <
boekewurm+postgres@gmail.com> wrote:

I don't think this is an actionable change, as this wastes 4 more bytes
(or 8 with alignment) in nearly all WAL records that don't use the
HEAP/HEAP2/XLOG rmgrs, which would then be up to 10 (if not 14, when
64but-aligned) bytes per record. Unless something like [0] gets committed
this will add a significant write overhead to all operations, even if they
are not doing anything that needs an XID.

Also, I don't think we need to support transactions that stay alive and
change things for longer than 2^31 concurrently created transactions, so we
could well add a fxid to each WAL segment header (and checkpoint record?)
and calculate the fxid of each record as a relative fxid off of that.

Thank you for your reply. Yes, this is a good idea. Actually, this is

exactly what I was trying to do at first.
But in this case, we have to add more locks on TransamVariables->nextXid,
and I've abandoned the idea.
Maybe, I should look in this direction.

On Sun, 31 Dec 2023 at 03:44, Michael Paquier <michael@paquier.xyz> wrote:

And FWIW, echoing with Matthias, making these generic structures
arbitrary larger is a non-starter. We should try to make them
shorter.

Yeah, obviously, this is patch make WAL bigger. I'll try to look into the
idea of fxid calculation, as mentioned above.
It might in part be a "chicken and the egg" situation. It is very hard to
split overall 64xid patch into smaller pieces
with each part been a meaningful and beneficial for current 32xids Postgres.

Anyway, thanks for reply, appreciate it.

--
Best regards,
Maxim Orlov.

On Mon, Jan 1, 2024 at 1:15 AM Maxim Orlov <orlovmg@gmail.com> wrote:

Yeah, obviously, this is patch make WAL bigger. I'll try to look into the idea of fxid calculation, as mentioned above.
It might in part be a "chicken and the egg" situation. It is very hard to split overall 64xid patch into smaller pieces
with each part been a meaningful and beneficial for current 32xids Postgres.

I agree, but I think that's what we need to try to do. I mean, we
don't want to commit patches that individually make things *worse* on
the theory that when we do that enough times the result will be
overall better. And even if the individual patches seem to be entirely
neutral, that still doesn't offer great hope that the final result
will be an improvement.

Personally, I'm not convinced that "use 64-bit XIDs everywhere" is a
desirable goal. The space consumption issue here is just the tip of
the iceberg, and really exists in many places where we store XIDs. We
don't want to make tuple headers wider any more than we want to bloat
the WAL. We don't want to make the ProcArray bigger, either, not
because of memory consumption but because of the speed of memory
access. What we really want to do is fix the practical problems that
32-bit XIDs cause. AFAIK there are basically three such problems:

1. SLRUs that are indexed by XID can wrap around, or try to, leading
to confusion and bugs in the code and maybe errors when something
fills up.

2. If a table's relfrozenxid become older than ~2B, we have to stop
XID assignment until the problem is cured.

3. If a running transaction's XID age exceeds ~2B, we can't start new
transactions until the problem is cured.

There's already a patch for (1), I believe. I think we can cure that
by indexing SLRUs by 64-bit integers without changing how XIDs are
stored outside of SLRUs. Various people have attempted (2), for
example with an XID-per-page approach, but nothing's been committed
yet. We can't attempt to cure (3) until (1) or (2) are fixed, and I
think that would require using 64-bit XIDs in the ProcArray, but note
that (3) is less serious: (1) and (2) constrain the size of the XID
range that can legally appear on disk, whereas (3) only constraints
the size of the XID range that can legally occur in memory. That makes
a big difference, because reducing the size of the XID range that can
legally appear on disk requires vacuuming all tables with older
relfrozenxids which in the worst case takes time proportional to the
disk utilization of the instance, whereas reducing the size of the XID
range that can legally appear in memory just requires ending
transactions (including possibly prepared transactions) and removing
replication slots, which can be done in O(1) time.

Maybe this analysis I've just given isn't quite right, but my point is
that we should try to think hard about where in the system 32-bit XIDs
suck and for what reason, and use that as a guide to what to change
first.

--
Robert Haas
EDB: http://www.enterprisedb.com

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">On 1/2/24 1:58 PM, Robert Haas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CA+TgmoYRg1MDF6a-QDrccVO=d2S37bjz54=xSzwen9FYHqw=7Q@mail.gmail.com">
<pre>Maybe this analysis I've just given isn't quite right, but my point is
that we should try to think hard about where in the system 32-bit XIDs
suck and for what reason, and use that as a guide to what to change
first.</pre>
</blockquote>
<p>Very much this. The biggest reason 2B XIDs are such an issue is
because it's incredibly expensive to move the window forward,
which is governed by on-disk stuff.<br>
</p>
</body>
</html>

Hi, Maxim, and Happy New Year!

On Mon, 1 Jan 2024 at 10:15, Maxim Orlov <orlovmg@gmail.com> wrote:

On Fri, 29 Dec 2023 at 16:36, Matthias van de Meent <
boekewurm+postgres@gmail.com> wrote:

I don't think this is an actionable change, as this wastes 4 more bytes
(or 8 with alignment) in nearly all WAL records that don't use the
HEAP/HEAP2/XLOG rmgrs, which would then be up to 10 (if not 14, when
64but-aligned) bytes per record. Unless something like [0] gets committed
this will add a significant write overhead to all operations, even if they
are not doing anything that needs an XID.

Also, I don't think we need to support transactions that stay alive and
change things for longer than 2^31 concurrently created transactions, so we
could well add a fxid to each WAL segment header (and checkpoint record?)
and calculate the fxid of each record as a relative fxid off of that.

Thank you for your reply. Yes, this is a good idea. Actually, this is

exactly what I was trying to do at first.
But in this case, we have to add more locks on TransamVariables->nextXid,
and I've abandoned the idea.
Maybe, I should look in this direction.

On Sun, 31 Dec 2023 at 03:44, Michael Paquier <michael@paquier.xyz> wrote:

And FWIW, echoing with Matthias, making these generic structures
arbitrary larger is a non-starter. We should try to make them
shorter.

Yeah, obviously, this is patch make WAL bigger. I'll try to look into the
idea of fxid calculation, as mentioned above.
It might in part be a "chicken and the egg" situation. It is very hard to
split overall 64xid patch into smaller pieces
with each part been a meaningful and beneficial for current 32xids
Postgres.

The discussion quoted by you as OP [0]/messages/by-id/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com shows that there is a wide range of
opinions on whether we need 64-bit XIDs and what is the better way to
implement it. We can also continue discussing implementation details in
that thread [0]/messages/by-id/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com.

I agree that the step towards something big should not make things worse
just now. Does increasing the WAL header for several bytes make performance
change detectable at all? Maybe it's worth simulating a workload with a
large amount of WAL records and seeing how it works. I think it's regarding
this patchset only and could remove or substantiate the main questions
about the current patchset.

[0]: /messages/by-id/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com
/messages/by-id/CACG=ezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe=pyyjVWA@mail.gmail.com

Kind regards,
Pavel Borisov.