#XX000: ERROR: tuple concurrently updated
Hi. A tester just tried to restore two custom backups (not official
PostgreSQL ones) concurrently.
The restore process creates a new schema, some roles,and does some
role-to-role grants, before loading the data.
The two restores are independent, i.e. create different non-overlapping
schemas.
But both are "registered" in the same pre-existing schema, SCH1.
And both run within a single transaction, that does everything, DDLs and
DMLs.
The first session was doing a longer bigger restore, and was started first.
While the second session was doing a faster shorter one, and started a few
seconds later, concurrently.
The second session completed OK.
But the first session errors out with:
Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
"SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
Thus I'm trying to understand what's going on.
As noted above, SCH1 is the pre-existing schema, while SCH2 is the new one
created by the restore.
The second session that completes normally did a `GRANT USAGE ON SCHEMA
"SCH1", "SCH3" TO "SCH3:RO", "SCH3:RW", "SCH3:SU"` within its own
transaction.
When I research #XX000, it often mentions doing a SELECT FOR UPDATE, but
here I'm doing GRANTs, so I don't think that applies.
Is the issue related to trying to change SCHEMA ACLs for SCH1 concurrently,
in two long running transactions? How am I supposed to resolve this?
I have to confess being a little surprised and blindsided by this issue.
Help understanding what's going on, and how to fix it, would be greatly
appreciated.
Thanks, --DD
Dominique Devienne <ddevienne@gmail.com> writes:
Hi. A tester just tried to restore two custom backups (not official
PostgreSQL ones) concurrently.
...
The second session completed OK.
But the first session errors out with:
Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
"SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updated
Thus I'm trying to understand what's going on.
Since both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
Is the issue related to trying to change SCHEMA ACLs for SCH1 concurrently,
in two long running transactions? How am I supposed to resolve this?
The window is probably too small to hit if each restore is committing
as it goes, but if you run in --single-transaction mode then this
isn't surprising. I'd say don't try to run concurrent restores.
regards, tom lane
On Thu, Feb 20, 2025 at 4:27 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dominique Devienne <ddevienne@gmail.com> writes:
Hi. A tester just tried to restore two custom backups (not official
PostgreSQL ones) concurrently.
...
The second session completed OK.
But the first session errors out with:Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
"SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updatedThus I'm trying to understand what's going on.
Since both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
Thanks for confirming Tom.
Is the issue related to trying to change SCHEMA ACLs for SCH1
concurrently,
in two long running transactions? How am I supposed to resolve this?
The window is probably too small to hit if each restore is committing
as it goes, but if you run in --single-transaction mode then this
isn't surprising. I'd say don't try to run concurrent restores.
First, I'm not in psql, but my own code. And as mentioned, I'm already in
the equivalent,
doing everything (DDLs and DMLs) in a single transaction.
Second, not doing concurrent restores is not an option.
So I need to separate operations related to SCH2,3,..., which are all
independent,
and SCH1 which is shared, as separate long running and short transactions,
respectively.
With perhaps some retry logic on the SCH1, just in case. --DD
Since you are willing to break the all one transaction rule, and if the
restores were created via pg_dump, you could use the --section argument to
split things up, run the "pre-data" sections serially, and the rest ("data"
and "post-data") concurrently.
--
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
On Thu, Feb 20, 2025 at 4:27 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dominique Devienne <ddevienne@gmail.com> writes:
Error: DDL Error: GRANT USAGE ON SCHEMA "SCH1", "SCH2" TO "SCH2:RO",
"SCH2:RW", "SCH2:SU": #XX000: ERROR: tuple concurrently updatedSince both restores tried to grant some permissions on SCH1, they
both had to update SCH1's pg_namespace row (specifically nspacl).
We have no support for concurrent updates in the catalog-manipulation
code, so if the second run arrives at that step before the first
one has committed its pg_namespace change, you get this error.
Hi Tom, and al.
I have a related question, on role-to-role grants this time.
Above, it was contention on pg_namespace.nspacl in two transactions.
But during those "restore" transactions, I must also make role-to-role
grants,
which AFAIK involve adding rows to pg_auth_members. So they are not subject
to the same "no support for concurrent updates in the catalog-manipulation"
you mentioned, as schema-to-role grants are, right? Because that's an
insert,
not an update? Just want to make sure, as I'm thinking how to change our
code.
Thanks, --DD
Dominique Devienne <ddevienne@gmail.com> writes:
But during those "restore" transactions, I must also make role-to-role
grants,
which AFAIK involve adding rows to pg_auth_members. So they are not subject
to the same "no support for concurrent updates in the catalog-manipulation"
you mentioned, as schema-to-role grants are, right? Because that's an
insert,
not an update? Just want to make sure, as I'm thinking how to change our
code.
Yeah, I think "GRANT role" should be relatively immune to that
problem, as long as you're making distinct grants (not same
grantor/grantee/granted roles). Wouldn't hurt to test.
regards, tom lane