Conflict Detection and Resolution
Hello hackers,
Please find the proposal for Conflict Detection and Resolution (CDR)
for Logical replication.
<Thanks to Nisha, Hou-San, and Amit who helped in figuring out the
below details.>
Introduction
================
In case the node is subscribed to multiple providers, or when local
writes happen on a subscriber, conflicts can arise for the incoming
changes. CDR is the mechanism to automatically detect and resolve
these conflicts depending on the application and configurations.
CDR is not applicable for the initial table sync. If locally, there
exists conflicting data on the table, the table sync worker will fail.
Please find the details on CDR in apply worker for INSERT, UPDATE and
DELETE operations:
INSERT
================
To resolve INSERT conflict on subscriber, it is important to find out
the conflicting row (if any) before we attempt an insertion. The
indexes or search preference for the same will be:
First check for replica identity (RI) index.
- if not found, check for the primary key (PK) index.
- if not found, then check for unique indexes (individual ones or
added by unique constraints)
- if unique index also not found, skip CDR
Note: if no RI index, PK, or unique index is found but
REPLICA_IDENTITY_FULL is defined, CDR will still be skipped.
The reason being that even though a row can be identified with
REPLICAT_IDENTITY_FULL, such tables are allowed to have duplicate
rows. Hence, we should not go for conflict detection in such a case.
In case of replica identity ‘nothing’ and in absence of any suitable
index (as defined above), CDR will be skipped for INSERT.
Conflict Type:
----------------
insert_exists: A conflict is detected when the table has the same
value for a key column as the new value in the incoming row.
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.
The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.
It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.
Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.
UPDATE
================
Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.
Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.
Conflict Resolutions:
----------------
a) latest_timestamp_wins: The change with later commit timestamp
wins. Can be used for ‘update_differ’.
b) earliest_timestamp_wins: The change with earlier commit
timestamp wins. Can be used for ‘update_differ’.
c) apply: The remote change is always applied. Can be used for
‘update_differ’.
d) apply_or_skip: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then the change is skipped. Can be used for
‘update_missing’ or ‘update_deleted’.
e) apply_or_error: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then error is raised. Can be used for
‘update_missing’ or ‘update_deleted’.
f) skip: Remote change is skipped and local one is retained. Can be
used for any conflict type.
g) error: Error out of conflict. Replication is stopped, manual
action is needed. Can be used for any conflict type.
To support UPDATE CDR, the presence of either replica identity Index
or primary key is required on target node. Update CDR will not be
supported in absence of replica identity index or primary key even
though REPLICA IDENTITY FULL is set. Please refer to "UPDATE" in
"Noteworthey Scenarios" section in [1]https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution for further details.
DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.
Conflict Resolutions:
----------------
a) error : Error out on conflict. Replication is stopped, manual
action is needed.
b) skip : The remote change is skipped.
Configuring Conflict Resolution:
------------------------------------------------
There are two parts when it comes to configuring CDR:
a) Enabling/Disabling conflict detection.
b) Configuring conflict resolvers for different conflict types.
Users can sometimes create multiple subscriptions on the same node,
subscribing to different tables to improve replication performance by
starting multiple apply workers. If the tables in one subscription are
less likely to cause conflict, then it is possible that user may want
conflict detection disabled for that subscription to avoid detection
latency while enabling it for other subscriptions. This generates a
requirement to make ‘conflict detection’ configurable per
subscription. While the conflict resolver configuration can remain
global. All the subscriptions which opt for ‘conflict detection’ will
follow global conflict resolver configuration.
To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.
To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
-------------------------
Apart from the above three main operations and resolver configuration,
there are more conflict types like primary-key updates, multiple
unique constraints etc and some special scenarios to be considered.
Complete design details can be found in [1]https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution.
[1]: https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution
thanks
Shveta
On 5/23/24 08:36, shveta malik wrote:
Hello hackers,
Please find the proposal for Conflict Detection and Resolution (CDR)
for Logical replication.
<Thanks to Nisha, Hou-San, and Amit who helped in figuring out the
below details.>Introduction
================
In case the node is subscribed to multiple providers, or when local
writes happen on a subscriber, conflicts can arise for the incoming
changes. CDR is the mechanism to automatically detect and resolve
these conflicts depending on the application and configurations.
CDR is not applicable for the initial table sync. If locally, there
exists conflicting data on the table, the table sync worker will fail.
Please find the details on CDR in apply worker for INSERT, UPDATE and
DELETE operations:
Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.
Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.
INSERT
================
To resolve INSERT conflict on subscriber, it is important to find out
the conflicting row (if any) before we attempt an insertion. The
indexes or search preference for the same will be:
First check for replica identity (RI) index.
- if not found, check for the primary key (PK) index.
- if not found, then check for unique indexes (individual ones or
added by unique constraints)
- if unique index also not found, skip CDRNote: if no RI index, PK, or unique index is found but
REPLICA_IDENTITY_FULL is defined, CDR will still be skipped.
The reason being that even though a row can be identified with
REPLICAT_IDENTITY_FULL, such tables are allowed to have duplicate
rows. Hence, we should not go for conflict detection in such a case.
It's not clear to me why would REPLICA_IDENTITY_FULL mean the table is
allowed to have duplicate values? It just means the upstream is sending
the whole original row, there can still be a PK/UNIQUE index on both the
publisher and subscriber.
In case of replica identity ‘nothing’ and in absence of any suitable
index (as defined above), CDR will be skipped for INSERT.Conflict Type:
----------------
insert_exists: A conflict is detected when the table has the same
value for a key column as the new value in the incoming row.Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.
Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?
The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.
How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.
UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.
I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?
Conflict Resolutions:
----------------
a) latest_timestamp_wins: The change with later commit timestamp
wins. Can be used for ‘update_differ’.
b) earliest_timestamp_wins: The change with earlier commit
timestamp wins. Can be used for ‘update_differ’.
c) apply: The remote change is always applied. Can be used for
‘update_differ’.
d) apply_or_skip: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then the change is skipped. Can be used for
‘update_missing’ or ‘update_deleted’.
e) apply_or_error: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then error is raised. Can be used for
‘update_missing’ or ‘update_deleted’.
f) skip: Remote change is skipped and local one is retained. Can be
used for any conflict type.
g) error: Error out of conflict. Replication is stopped, manual
action is needed. Can be used for any conflict type.To support UPDATE CDR, the presence of either replica identity Index
or primary key is required on target node. Update CDR will not be
supported in absence of replica identity index or primary key even
though REPLICA IDENTITY FULL is set. Please refer to "UPDATE" in
"Noteworthey Scenarios" section in [1] for further details.DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.Conflict Resolutions:
----------------
a) error : Error out on conflict. Replication is stopped, manual
action is needed.
b) skip : The remote change is skipped.Configuring Conflict Resolution:
------------------------------------------------
There are two parts when it comes to configuring CDR:a) Enabling/Disabling conflict detection.
b) Configuring conflict resolvers for different conflict types.Users can sometimes create multiple subscriptions on the same node,
subscribing to different tables to improve replication performance by
starting multiple apply workers. If the tables in one subscription are
less likely to cause conflict, then it is possible that user may want
conflict detection disabled for that subscription to avoid detection
latency while enabling it for other subscriptions. This generates a
requirement to make ‘conflict detection’ configurable per
subscription. While the conflict resolver configuration can remain
global. All the subscriptions which opt for ‘conflict detection’ will
follow global conflict resolver configuration.To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
I very much doubt we want a single global conflict resolver, or even one
resolver per subscription. It seems like a very table-specific thing.
Also, doesn't all this whole design ignore the concurrency between
publishers? Isn't this problematic considering the commit timestamps may
go backwards (for a given publisher), which means the conflict
resolution is not deterministic (as it depends on how exactly it
interleaves)?
-------------------------
Apart from the above three main operations and resolver configuration,
there are more conflict types like primary-key updates, multiple
unique constraints etc and some special scenarios to be considered.
Complete design details can be found in [1].[1]: https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution
Hmmm, not sure it's good to have a "complete" design on wiki, and only
some subset posted to the mailing list. I haven't compared what the
differences are, though.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 5/23/24 08:36, shveta malik wrote:
Hello hackers,
Please find the proposal for Conflict Detection and Resolution (CDR)
for Logical replication.
<Thanks to Nisha, Hou-San, and Amit who helped in figuring out the
below details.>Introduction
================
In case the node is subscribed to multiple providers, or when local
writes happen on a subscriber, conflicts can arise for the incoming
changes. CDR is the mechanism to automatically detect and resolve
these conflicts depending on the application and configurations.
CDR is not applicable for the initial table sync. If locally, there
exists conflicting data on the table, the table sync worker will fail.
Please find the details on CDR in apply worker for INSERT, UPDATE and
DELETE operations:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.
Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.
Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.
Can you please explain a little bit more on this.
INSERT
================
To resolve INSERT conflict on subscriber, it is important to find out
the conflicting row (if any) before we attempt an insertion. The
indexes or search preference for the same will be:
First check for replica identity (RI) index.
- if not found, check for the primary key (PK) index.
- if not found, then check for unique indexes (individual ones or
added by unique constraints)
- if unique index also not found, skip CDRNote: if no RI index, PK, or unique index is found but
REPLICA_IDENTITY_FULL is defined, CDR will still be skipped.
The reason being that even though a row can be identified with
REPLICAT_IDENTITY_FULL, such tables are allowed to have duplicate
rows. Hence, we should not go for conflict detection in such a case.It's not clear to me why would REPLICA_IDENTITY_FULL mean the table is
allowed to have duplicate values? It just means the upstream is sending
the whole original row, there can still be a PK/UNIQUE index on both the
publisher and subscriber.
Yes, right. Sorry for confusion. I meant the same i.e. in absence of
'RI index, PK, or unique index', tables can have duplicates. So even
in presence of Replica-identity (FULL in this case) but in absence of
unique/primary index, CDR will be skipped for INSERT.
In case of replica identity ‘nothing’ and in absence of any suitable
index (as defined above), CDR will be skipped for INSERT.Conflict Type:
----------------
insert_exists: A conflict is detected when the table has the same
value for a key column as the new value in the incoming row.Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?
Initially, for the sake of simplicity, we are targeting to support
built-in resolvers. But we have a plan to work on user-defined
resolvers as well. We shall propose that separately.
The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.
Are you pointing to the issue where a session/txn has taken
'xactStopTimestamp' timestamp earlier but is delayed to insert record
in XLOG, while another session/txn which has taken timestamp slightly
later succeeded to insert the record IN XLOG sooner than the session1,
making LSN and Timestamps out of sync? Going by this scenario, the
commit-timestamp may not be reflective of actual commits and thus
timestamp-based resolvers may take wrong decisions. Or do you mean
something else?
If this is the problem you are referring to, then I think this needs a
fix at the publisher side. Let me think more about it . Kindly let me
know if you have ideas on how to tackle it.
UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?
Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.
Regarding "TOAST column" for deleted row cases, we may need to dig
more. Thanks for bringing this case. Let me analyze more here.
Conflict Resolutions:
----------------
a) latest_timestamp_wins: The change with later commit timestamp
wins. Can be used for ‘update_differ’.
b) earliest_timestamp_wins: The change with earlier commit
timestamp wins. Can be used for ‘update_differ’.
c) apply: The remote change is always applied. Can be used for
‘update_differ’.
d) apply_or_skip: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then the change is skipped. Can be used for
‘update_missing’ or ‘update_deleted’.
e) apply_or_error: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then error is raised. Can be used for
‘update_missing’ or ‘update_deleted’.
f) skip: Remote change is skipped and local one is retained. Can be
used for any conflict type.
g) error: Error out of conflict. Replication is stopped, manual
action is needed. Can be used for any conflict type.To support UPDATE CDR, the presence of either replica identity Index
or primary key is required on target node. Update CDR will not be
supported in absence of replica identity index or primary key even
though REPLICA IDENTITY FULL is set. Please refer to "UPDATE" in
"Noteworthey Scenarios" section in [1] for further details.DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.Conflict Resolutions:
----------------
a) error : Error out on conflict. Replication is stopped, manual
action is needed.
b) skip : The remote change is skipped.Configuring Conflict Resolution:
------------------------------------------------
There are two parts when it comes to configuring CDR:a) Enabling/Disabling conflict detection.
b) Configuring conflict resolvers for different conflict types.Users can sometimes create multiple subscriptions on the same node,
subscribing to different tables to improve replication performance by
starting multiple apply workers. If the tables in one subscription are
less likely to cause conflict, then it is possible that user may want
conflict detection disabled for that subscription to avoid detection
latency while enabling it for other subscriptions. This generates a
requirement to make ‘conflict detection’ configurable per
subscription. While the conflict resolver configuration can remain
global. All the subscriptions which opt for ‘conflict detection’ will
follow global conflict resolver configuration.To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
I very much doubt we want a single global conflict resolver, or even one
resolver per subscription. It seems like a very table-specific thing.
Even we thought about this. We feel that even if we go for table based
or subscription based resolvers configuration, there may be use case
scenarios where the user is not interested in configuring resolvers
for each table and thus may want to give global ones. Thus, we should
provide a way for users to do global configuration. Thus we started
with global one. I have noted your point here and would also like to
know the opinion of others. We are open to discussion. We can either
opt for any of these 2 options (global or table) or we can opt for
both global and table/sub based one.
Also, doesn't all this whole design ignore the concurrency between
publishers? Isn't this problematic considering the commit timestamps may
go backwards (for a given publisher), which means the conflict
resolution is not deterministic (as it depends on how exactly it
interleaves)?-------------------------
Apart from the above three main operations and resolver configuration,
there are more conflict types like primary-key updates, multiple
unique constraints etc and some special scenarios to be considered.
Complete design details can be found in [1].[1]: https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution
Hmmm, not sure it's good to have a "complete" design on wiki, and only
some subset posted to the mailing list. I haven't compared what the
differences are, though.
It would have been difficult to mention all the details in email
(including examples and corner scenarios) and thus we thought that it
will be better to document everything in wiki page for the time being.
We can keep on discussing the design and all the scenarios on need
basis (before implementation phase of that part) and thus eventually
everything will come in email on hackers. With out first patch, we
plan to provide everything in a README as well.
thanks
Shveta
On Mon, May 27, 2024 at 11:19 AM shveta malik <shveta.malik@gmail.com> wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 5/23/24 08:36, shveta malik wrote:
Hello hackers,
Please find the proposal for Conflict Detection and Resolution (CDR)
for Logical replication.
<Thanks to Nisha, Hou-San, and Amit who helped in figuring out the
below details.>Introduction
================
In case the node is subscribed to multiple providers, or when local
writes happen on a subscriber, conflicts can arise for the incoming
changes. CDR is the mechanism to automatically detect and resolve
these conflicts depending on the application and configurations.
CDR is not applicable for the initial table sync. If locally, there
exists conflicting data on the table, the table sync worker will fail.
Please find the details on CDR in apply worker for INSERT, UPDATE and
DELETE operations:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.Can you please explain a little bit more on this.
INSERT
================
To resolve INSERT conflict on subscriber, it is important to find out
the conflicting row (if any) before we attempt an insertion. The
indexes or search preference for the same will be:
First check for replica identity (RI) index.
- if not found, check for the primary key (PK) index.
- if not found, then check for unique indexes (individual ones or
added by unique constraints)
- if unique index also not found, skip CDRNote: if no RI index, PK, or unique index is found but
REPLICA_IDENTITY_FULL is defined, CDR will still be skipped.
The reason being that even though a row can be identified with
REPLICAT_IDENTITY_FULL, such tables are allowed to have duplicate
rows. Hence, we should not go for conflict detection in such a case.It's not clear to me why would REPLICA_IDENTITY_FULL mean the table is
allowed to have duplicate values? It just means the upstream is sending
the whole original row, there can still be a PK/UNIQUE index on both the
publisher and subscriber.Yes, right. Sorry for confusion. I meant the same i.e. in absence of
'RI index, PK, or unique index', tables can have duplicates. So even
in presence of Replica-identity (FULL in this case) but in absence of
unique/primary index, CDR will be skipped for INSERT.In case of replica identity ‘nothing’ and in absence of any suitable
index (as defined above), CDR will be skipped for INSERT.Conflict Type:
----------------
insert_exists: A conflict is detected when the table has the same
value for a key column as the new value in the incoming row.Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?Initially, for the sake of simplicity, we are targeting to support
built-in resolvers. But we have a plan to work on user-defined
resolvers as well. We shall propose that separately.The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.Are you pointing to the issue where a session/txn has taken
'xactStopTimestamp' timestamp earlier but is delayed to insert record
in XLOG, while another session/txn which has taken timestamp slightly
later succeeded to insert the record IN XLOG sooner than the session1,
making LSN and Timestamps out of sync? Going by this scenario, the
commit-timestamp may not be reflective of actual commits and thus
timestamp-based resolvers may take wrong decisions. Or do you mean
something else?If this is the problem you are referring to, then I think this needs a
fix at the publisher side. Let me think more about it . Kindly let me
know if you have ideas on how to tackle it.UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.Regarding "TOAST column" for deleted row cases, we may need to dig
more. Thanks for bringing this case. Let me analyze more here.
I tested a simple case with a table with one TOAST column and found
that when a tuple with a TOAST column is deleted, both the tuple and
corresponding pg_toast entries are marked as ‘deleted’ (dead) but not
removed immediately. The main tuple and respective pg_toast entry are
permanently deleted only during vacuum. First, the main table’s dead
tuples are vacuumed, followed by the secondary TOAST relation ones (if
available).
Please let us know if you have a specific scenario in mind where the
TOAST column data is deleted immediately upon ‘delete’ operation,
rather than during vacuum, which we are missing.
Thanks,
Nisha
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 5/23/24 08:36, shveta malik wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.
One of the possible scenarios discussed at pgconf.dev with Tomas for
this was as follows:
Say there are two publisher nodes PN1, PN2, and subscriber node SN3.
The logical replication is configured such that a subscription on SN3
has publications from both PN1 and PN2. For example, SN3 (sub) -> PN1,
PN2 (p1, p2)
Now, on PN1, we have the following operations that update the same row:
T1
Update-1 on table t1 at LSN1 (1000) on time (200)
T2
Update-2 on table t1 at LSN2 (2000) on time (100)
Then in parallel, we have the following operation on node PN2 that
updates the same row as Update-1, and Update-2 on node PN1.
T3
Update-3 on table t1 at LSN(1500) on time (150)
By theory, we can have a different state on subscribers depending on
the order of updates arriving at SN3 which shouldn't happen. Say, the
order in which they reach SN3 is: Update-1, Update-2, Update-3 then
the final row we have is by Update-3 considering we have configured
last_update_wins as a conflict resolution method. Now, consider the
other order: Update-1, Update-3, Update-2, in this case, the final
row will be by Update-2 because when we try to apply Update-3, it will
generate a conflict and as per the resolution method
(last_update_wins) we need to retain Update-1.
On further thinking, the operations on node-1 PN-1 as defined above
seem impossible because one of the Updates needs to wait for the other
to write a commit record. So the commits may happen with LSN1 < LSN2
but with T1 > T2 but they can't be on the same row due to locks. So,
the order of apply should still be consistent. Am, I missing
something?
--
With Regards,
Amit Kapila.
On Mon, May 27, 2024 at 11:19 AM shveta malik <shveta.malik@gmail.com> wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.
Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.
I think to make 'update_deleted' work, we need another scan with a
different snapshot type to find the recently deleted row. I don't know
if it is a good idea to scan the index twice with different snapshots,
so for the sake of simplicity, can we consider 'updated_deleted' same
as 'update_missing'? If we think it is an important case to consider
then we can try to accomplish this once we finalize the
design/implementation of other resolution methods.
To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
I very much doubt we want a single global conflict resolver, or even one
resolver per subscription. It seems like a very table-specific thing.
+1 to make it a table-level configuration but we probably need
something at the global level as well such that by default if users
don't define anything at table-level global-level configuration will
be used.
Also, doesn't all this whole design ignore the concurrency between
publishers? Isn't this problematic considering the commit timestamps may
go backwards (for a given publisher), which means the conflict
resolution is not deterministic (as it depends on how exactly it
interleaves)?
I am not able to imagine the cases you are worried about. Can you
please be specific? Is it similar to the case I described in
yesterday's email [1]/messages/by-id/CAA4eK1JTMiBOoGqkt=aLPLU8Rs45ihbLhXaGHsz8XC76+OG3+Q@mail.gmail.com?
[1]: /messages/by-id/CAA4eK1JTMiBOoGqkt=aLPLU8Rs45ihbLhXaGHsz8XC76+OG3+Q@mail.gmail.com
--
With Regards,
Amit Kapila.
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.
I do not have the exact scenario for this. But I feel, if 2 nodes are
concurrently inserting different data against a primary key, then some
users may have preferences that retain the row which was inserted
earlier. It is no different from latest_timestamp_wins. It totally
depends upon what kind of application and requirement the user may
have, based on which, he may discard the later coming rows (specially
for INSERT case).
Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.I think to make 'update_deleted' work, we need another scan with a
different snapshot type to find the recently deleted row. I don't know
if it is a good idea to scan the index twice with different snapshots,
so for the sake of simplicity, can we consider 'updated_deleted' same
as 'update_missing'? If we think it is an important case to consider
then we can try to accomplish this once we finalize the
design/implementation of other resolution methods.
I think it is important for scenarios when data is being updated and
deleted concurrently. But yes, I agree that implementation may have
some performance hit for this case. We can tackle this scenario at a
later stage.
To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
I very much doubt we want a single global conflict resolver, or even one
resolver per subscription. It seems like a very table-specific thing.+1 to make it a table-level configuration but we probably need
something at the global level as well such that by default if users
don't define anything at table-level global-level configuration will
be used.Also, doesn't all this whole design ignore the concurrency between
publishers? Isn't this problematic considering the commit timestamps may
go backwards (for a given publisher), which means the conflict
resolution is not deterministic (as it depends on how exactly it
interleaves)?I am not able to imagine the cases you are worried about. Can you
please be specific? Is it similar to the case I described in
yesterday's email [1]?[1] - /messages/by-id/CAA4eK1JTMiBOoGqkt=aLPLU8Rs45ihbLhXaGHsz8XC76+OG3+Q@mail.gmail.com
thanks
Shveta
Hi,
This time at PGconf.dev[1]https://2024.pgconf.dev/, we had some discussions regarding this
project. The proposed approach is to split the work into two main
components. The first part focuses on conflict detection, which aims to
identify and report conflicts in logical replication. This feature will
enable users to monitor the unexpected conflicts that may occur. The
second part involves the actual conflict resolution. Here, we will provide
built-in resolutions for each conflict and allow user to choose which
resolution will be used for which conflict(as described in the initial
email of this thread).
Of course, we are open to alternative ideas and suggestions, and the
strategy above can be changed based on ongoing discussions and feedback
received.
Here is the patch of the first part work, which adds a new parameter
detect_conflict for CREATE and ALTER subscription commands. This new
parameter will decide if subscription will go for conflict detection. By
default, conflict detection will be off for a subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.
While there exist other conflict types in logical replication, such as an
incoming insert conflicting with an existing row due to a primary key or
unique index, these cases already result in constraint violation errors.
Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.
Best Regards,
Hou zj
Attachments:
v1-0001-Detect-update-and-delete-conflict-in-logical-repl.patchapplication/octet-stream; name=v1-0001-Detect-update-and-delete-conflict-in-logical-repl.patchDownload
From ac8da6fa047037ba1c2f6514e3481ce2afce0fea Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v1] Detect update and delete conflict in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.
While there exist other conflict types in logical replication, such as an
incoming insert conflicting with an existing row due to a primary key or
unique index, these cases already result in constraint violation errors.
Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.
---
doc/src/sgml/catalogs.sgml | 9 ++
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 43 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/replication/logical/worker.c | 145 ++++++++++++++---
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/test/regress/expected/subscription.out | 176 ++++++++++++---------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 +++---
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 419 insertions(+), 141 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 15f6255d86..495a6ea479 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8038,6 +8038,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..45f71d6386 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 53047cab5f..49ac738f26 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1359,7 +1359,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..80d6d02ecb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -147,6 +147,7 @@
#include <sys/stat.h>
#include <unistd.h>
+#include "access/commit_ts.h"
#include "access/table.h"
#include "access/tableam.h"
#include "access/twophase.h"
@@ -274,6 +275,31 @@ typedef enum
TRANS_PARALLEL_APPLY,
} TransApplyAction;
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ *
+ * For now, this list includes conflict types that will prompt additional logging
+ * only if conflict detection is turned on. Other conflicts that already
+ * lead to constraint violation errors are excluded from this enumeration.
+ */
+typedef enum
+{
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+const char *const ConflictTypeNames[] = {
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
/* errcontext tracker */
ApplyErrorCallbackArg apply_error_callback_arg =
{
@@ -416,6 +442,14 @@ static inline void reset_apply_error_context_info(void);
static TransApplyAction get_transaction_apply_action(TransactionId xid,
ParallelApplyWorkerInfo **winfo);
+static bool get_tuple_commit_ts(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin,
+ TimestampTz *localts);
+static void report_apply_conflict(ConflictType type, Relation localrel,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts);
+
/*
* Form the origin name for the subscription.
*
@@ -2664,6 +2698,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ get_tuple_commit_ts(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ report_apply_conflict(CT_UPDATE_DIFFER, localrel, localxmin,
+ localorigin, localts);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2729,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ report_apply_conflict(CT_UPDATE_MISSING, localrel,
+ InvalidTransactionId, InvalidRepOriginId, 0);
}
/* Cleanup. */
@@ -2821,13 +2866,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ report_apply_conflict(CT_DELETE_MISSING, localrel,
+ InvalidTransactionId, InvalidRepOriginId, 0);
}
/* Cleanup. */
@@ -3005,13 +3047,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ report_apply_conflict(CT_UPDATE_MISSING,
+ partrel,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
return;
}
@@ -5088,3 +5130,70 @@ get_transaction_apply_action(TransactionId xid, ParallelApplyWorkerInfo **winfo)
return TRANS_LEADER_APPLY;
}
}
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Returns true if the commit timestamp data was found, false otherwise.
+ */
+static bool
+get_tuple_commit_ts(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+static void
+report_apply_conflict(ConflictType type, Relation localrel,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts)
+{
+ switch (type)
+ {
+ case CT_UPDATE_DIFFER:
+ ereport(LOG,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s\"",
+ ConflictTypeNames[type], RelationGetRelationName(localrel)),
+ errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts)));
+ break;
+ case CT_UPDATE_MISSING:
+ ereport(LOG,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s\"",
+ ConflictTypeNames[type], RelationGetRelationName(localrel)),
+ errdetail("Did not find the row to be updated."));
+ break;
+ case CT_DELETE_MISSING:
+ ereport(LOG,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s\"",
+ ConflictTypeNames[type], RelationGetRelationName(localrel)),
+ errdetail("Did not find the row to be deleted."));
+ break;
+ }
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..c6b67c692d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5140,6 +5150,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..02aa4a6f32 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..0472fe2e87 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6597,6 +6597,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..aad4907d43 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..d74f6bdabe 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..02afbc2ed4 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index d427a1c16a..3b12be42ad 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.30.0.windows.2
On Wed, Jun 5, 2024 at 9:12 AM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.I do not have the exact scenario for this. But I feel, if 2 nodes are
concurrently inserting different data against a primary key, then some
users may have preferences that retain the row which was inserted
earlier. It is no different from latest_timestamp_wins. It totally
depends upon what kind of application and requirement the user may
have, based on which, he may discard the later coming rows (specially
for INSERT case).
I haven't read the complete design yet, but have we discussed how we
plan to deal with clock drift if we use timestamp-based conflict
resolution? For example, a user might insert something conflicting on
node1 first and then on node2. However, due to clock drift, the
timestamp from node2 might appear earlier. In this case, if we choose
"earliest timestamp wins," we would keep the changes from node2.
I haven't fully considered if this would cause any problems, but users
might detect this issue. For instance, a client machine might send a
change to node1 first and then, upon confirmation, send it to node2.
If the clocks on node1 and node2 are not synchronized, the changes
might appear in a different order. Does this seem like a potential
problem?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.
I can not think of a use case exactly in this context but it's very
common to have such a use case while designing a distributed
application with multiple clients. For example, when we are doing git
push concurrently from multiple clients it is expected that the
earliest commit wins.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 5, 2024 at 7:29 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.I can not think of a use case exactly in this context but it's very
common to have such a use case while designing a distributed
application with multiple clients. For example, when we are doing git
push concurrently from multiple clients it is expected that the
earliest commit wins.
Okay, I think it mostly boils down to something like what Shveta
mentioned where Inserts for a primary key can use
"earliest_timestamp_wins" resolution method [1]/messages/by-id/CAJpy0uC4riK8e6hQt8jcU+nXYmRRjnbFEapYNbmxVYjENxTw2g@mail.gmail.com. So, it seems useful
to support this method as well.
[1]: /messages/by-id/CAJpy0uC4riK8e6hQt8jcU+nXYmRRjnbFEapYNbmxVYjENxTw2g@mail.gmail.com
--
With Regards,
Amit Kapila.
On Thu, Jun 6, 2024 at 3:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 5, 2024 at 7:29 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.I can not think of a use case exactly in this context but it's very
common to have such a use case while designing a distributed
application with multiple clients. For example, when we are doing git
push concurrently from multiple clients it is expected that the
earliest commit wins.Okay, I think it mostly boils down to something like what Shveta
mentioned where Inserts for a primary key can use
"earliest_timestamp_wins" resolution method [1]. So, it seems useful
to support this method as well.
Correct, but we still need to think about how to make it work
correctly in the presence of a clock skew as I mentioned in one of my
previous emails.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 5, 2024 at 7:29 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 4, 2024 at 9:37 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
Can you share the use case of "earliest_timestamp_wins" resolution
method? It seems after the initial update on the local node, it will
never allow remote update to succeed which sounds a bit odd. Jan has
shared this and similar concerns about this resolution method, so I
have added him to the email as well.I can not think of a use case exactly in this context but it's very
common to have such a use case while designing a distributed
application with multiple clients. For example, when we are doing git
push concurrently from multiple clients it is expected that the
earliest commit wins.
Here are more use cases of the "earliest_timestamp_wins" resolution method:
1) Applications where the record of first occurrence of an event is
important. For example, sensor based applications like earthquake
detection systems, capturing the first seismic wave's time is crucial.
2) Scheduling systems, like appointment booking, prioritize the
earliest request when handling concurrent ones.
3) In contexts where maintaining chronological order is important -
a) Social media platforms display comments ensuring that the
earliest ones are visible first.
b) Finance transaction processing systems rely on timestamps to
prioritize the processing of transactions, ensuring that the earliest
transaction is handled first
--
Thanks,
Nisha
On Thu, Jun 6, 2024 at 5:16 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Here are more use cases of the "earliest_timestamp_wins" resolution method:
1) Applications where the record of first occurrence of an event is
important. For example, sensor based applications like earthquake
detection systems, capturing the first seismic wave's time is crucial.
2) Scheduling systems, like appointment booking, prioritize the
earliest request when handling concurrent ones.
3) In contexts where maintaining chronological order is important -
a) Social media platforms display comments ensuring that the
earliest ones are visible first.
b) Finance transaction processing systems rely on timestamps to
prioritize the processing of transactions, ensuring that the earliest
transaction is handled first
Thanks for sharing examples. However, these scenarios would be handled by
the application and not during replication. What we are discussing here is
the timestamp when a row was updated/inserted/deleted (or rather when the
transaction that updated row committed/became visible) and not a DML on
column which is of type timestamp. Some implementations use a hidden
timestamp column but that's different from a user column which captures
timestamp of (say) an event. The conflict resolution will be based on the
timestamp when that column's value was recorded in the database which may
be different from the value of the column itself.
If we use the transaction commit timestamp as basis for resolution, a
transaction where multiple rows conflict may end up with different rows
affected by that transaction being resolved differently. Say three
transactions T1, T2 and T3 on separate origins with timestamps t1, t2, and
t3 respectively changed rows r1, r2 and r2, r3 and r1, r4 respectively.
Changes to r1 and r2 will conflict. Let's say T2 and T3 are applied first
and then T1 is applied. If t2 < t1 < t3, r1 will end up with version of T3
and r2 will end up with version of T1 after applying all the three
transactions. Would that introduce an inconsistency between r1 and r2?
--
Best Wishes,
Ashutosh Bapat
On 5/27/24 07:48, shveta malik wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 5/23/24 08:36, shveta malik wrote:
Hello hackers,
Please find the proposal for Conflict Detection and Resolution (CDR)
for Logical replication.
<Thanks to Nisha, Hou-San, and Amit who helped in figuring out the
below details.>Introduction
================
In case the node is subscribed to multiple providers, or when local
writes happen on a subscriber, conflicts can arise for the incoming
changes. CDR is the mechanism to automatically detect and resolve
these conflicts depending on the application and configurations.
CDR is not applicable for the initial table sync. If locally, there
exists conflicting data on the table, the table sync worker will fail.
Please find the details on CDR in apply worker for INSERT, UPDATE and
DELETE operations:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.Can you please explain a little bit more on this.
I was referring to the well established consistency models / isolation
levels, e.g. READ COMMITTED or SNAPSHOT ISOLATION. This determines what
guarantees the application developer can expect, what anomalies can
happen, etc.
I don't think any such isolation level can be implemented with a simple
conflict resolution methods like last-update-wins etc. For example,
consider an active-active where both nodes do
UPDATE accounts SET balance=balance+1000 WHERE id=1
This will inevitably lead to a conflict, and while the last-update-wins
resolves this "consistently" on both nodes (e.g. ending with the same
result), it's essentially a lost update.
This is a very simplistic example of course, I recall there are various
more complex examples involving foreign keys, multi-table transactions,
constraints, etc. But in principle it's a manifestation of the same
inherent limitation of conflict detection and resolution etc.
Similarly, I believe this affects not just active-active, but also the
case where one node aggregates data from multiple publishers. Maybe not
to the same extent / it might be fine for that use case, but you said
the end goal is to use this for active-active. So I'm wondering what's
the plan, there.
If I'm writing an application for active-active using this conflict
handling, what assumptions can I make? Will Can I just do stuff as if on
a single node, or do I need to be super conscious about the zillion ways
things can misbehave in a distributed system?
My personal opinion is that the closer this will be to the regular
consistency levels, the better. If past experience taught me anything,
it's very hard to predict how distributed systems with eventual
consistency behave, and even harder to actually test the application in
such environment.
In any case, if there are any differences compared to the usual
behavior, it needs to be very clearly explained in the docs.
INSERT
================
To resolve INSERT conflict on subscriber, it is important to find out
the conflicting row (if any) before we attempt an insertion. The
indexes or search preference for the same will be:
First check for replica identity (RI) index.
- if not found, check for the primary key (PK) index.
- if not found, then check for unique indexes (individual ones or
added by unique constraints)
- if unique index also not found, skip CDRNote: if no RI index, PK, or unique index is found but
REPLICA_IDENTITY_FULL is defined, CDR will still be skipped.
The reason being that even though a row can be identified with
REPLICAT_IDENTITY_FULL, such tables are allowed to have duplicate
rows. Hence, we should not go for conflict detection in such a case.It's not clear to me why would REPLICA_IDENTITY_FULL mean the table is
allowed to have duplicate values? It just means the upstream is sending
the whole original row, there can still be a PK/UNIQUE index on both the
publisher and subscriber.Yes, right. Sorry for confusion. I meant the same i.e. in absence of
'RI index, PK, or unique index', tables can have duplicates. So even
in presence of Replica-identity (FULL in this case) but in absence of
unique/primary index, CDR will be skipped for INSERT.In case of replica identity ‘nothing’ and in absence of any suitable
index (as defined above), CDR will be skipped for INSERT.Conflict Type:
----------------
insert_exists: A conflict is detected when the table has the same
value for a key column as the new value in the incoming row.Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?Initially, for the sake of simplicity, we are targeting to support
built-in resolvers. But we have a plan to work on user-defined
resolvers as well. We shall propose that separately.The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.Are you pointing to the issue where a session/txn has taken
'xactStopTimestamp' timestamp earlier but is delayed to insert record
in XLOG, while another session/txn which has taken timestamp slightly
later succeeded to insert the record IN XLOG sooner than the session1,
making LSN and Timestamps out of sync? Going by this scenario, the
commit-timestamp may not be reflective of actual commits and thus
timestamp-based resolvers may take wrong decisions. Or do you mean
something else?If this is the problem you are referring to, then I think this needs a
fix at the publisher side. Let me think more about it . Kindly let me
know if you have ideas on how to tackle it.
Yes, this is the issue I'm talking about. We're acquiring the timestamp
when not holding the lock to reserve space in WAL, so the LSN and the
commit LSN may not actually correlate.
Consider this example I discussed with Amit last week:
node A:
XACT1: UPDATE t SET v = 1; LSN1 / T1
XACT2: UPDATE t SET v = 2; LSN2 / T2
node B
XACT3: UPDATE t SET v = 3; LSN3 / T3
And assume LSN1 < LSN2, T1 > T2 (i.e. the commit timestamp inversion),
and T2 < T3 < T1. Now consider that the messages may arrive in different
orders, due to async replication. Unfortunately, this would lead to
different results of the conflict resolution:
XACT1 - XACT2 - XACT3 => v=3 (T3 wins)
XACT3 - XACT1 - XACT2 => v=2 (T2 wins)
Now, I realize there's a flaw in this example - the (T1 > T2) inversion
can't actually happen, because these transactions have a dependency, and
thus won't commit concurrently. XACT1 will complete the commit, because
XACT2 starts to commit. And with monotonic clock (which is a requirement
for any timestamp-based resolution), that should guarantee (T1 < T2).
However, I doubt this is sufficient to declare victory. It's more likely
that there still are problems, but the examples are likely more complex
(changes to multiple tables, etc.).
I vaguely remember there were more issues with timestamp inversion, but
those might have been related to parallel apply etc.
UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.
My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.
But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.
Regarding "TOAST column" for deleted row cases, we may need to dig
more. Thanks for bringing this case. Let me analyze more here.Conflict Resolutions:
----------------
a) latest_timestamp_wins: The change with later commit timestamp
wins. Can be used for ‘update_differ’.
b) earliest_timestamp_wins: The change with earlier commit
timestamp wins. Can be used for ‘update_differ’.
c) apply: The remote change is always applied. Can be used for
‘update_differ’.
d) apply_or_skip: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then the change is skipped. Can be used for
‘update_missing’ or ‘update_deleted’.
e) apply_or_error: Remote change is converted to INSERT and is
applied. If the complete row cannot be constructed from the info
provided by the publisher, then error is raised. Can be used for
‘update_missing’ or ‘update_deleted’.
f) skip: Remote change is skipped and local one is retained. Can be
used for any conflict type.
g) error: Error out of conflict. Replication is stopped, manual
action is needed. Can be used for any conflict type.To support UPDATE CDR, the presence of either replica identity Index
or primary key is required on target node. Update CDR will not be
supported in absence of replica identity index or primary key even
though REPLICA IDENTITY FULL is set. Please refer to "UPDATE" in
"Noteworthey Scenarios" section in [1] for further details.DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.Conflict Resolutions:
----------------
a) error : Error out on conflict. Replication is stopped, manual
action is needed.
b) skip : The remote change is skipped.Configuring Conflict Resolution:
------------------------------------------------
There are two parts when it comes to configuring CDR:a) Enabling/Disabling conflict detection.
b) Configuring conflict resolvers for different conflict types.Users can sometimes create multiple subscriptions on the same node,
subscribing to different tables to improve replication performance by
starting multiple apply workers. If the tables in one subscription are
less likely to cause conflict, then it is possible that user may want
conflict detection disabled for that subscription to avoid detection
latency while enabling it for other subscriptions. This generates a
requirement to make ‘conflict detection’ configurable per
subscription. While the conflict resolver configuration can remain
global. All the subscriptions which opt for ‘conflict detection’ will
follow global conflict resolver configuration.To implement the above, subscription commands will be changed to have
one more parameter 'conflict_resolution=on/off', default will be OFF.To configure global resolvers, new DDL command will be introduced:
CONFLICT RESOLVER ON <conflict_type> IS <conflict_resolver>
I very much doubt we want a single global conflict resolver, or even one
resolver per subscription. It seems like a very table-specific thing.Even we thought about this. We feel that even if we go for table based
or subscription based resolvers configuration, there may be use case
scenarios where the user is not interested in configuring resolvers
for each table and thus may want to give global ones. Thus, we should
provide a way for users to do global configuration. Thus we started
with global one. I have noted your point here and would also like to
know the opinion of others. We are open to discussion. We can either
opt for any of these 2 options (global or table) or we can opt for
both global and table/sub based one.
I have no problem with a default / global conflict handler, as long as
there's a way to override this per table. This is especially important
for cases with custom conflict handler at table / column level.
Also, doesn't all this whole design ignore the concurrency between
publishers? Isn't this problematic considering the commit timestamps may
go backwards (for a given publisher), which means the conflict
resolution is not deterministic (as it depends on how exactly it
interleaves)?-------------------------
Apart from the above three main operations and resolver configuration,
there are more conflict types like primary-key updates, multiple
unique constraints etc and some special scenarios to be considered.
Complete design details can be found in [1].[1]: https://wiki.postgresql.org/wiki/Conflict_Detection_and_Resolution
Hmmm, not sure it's good to have a "complete" design on wiki, and only
some subset posted to the mailing list. I haven't compared what the
differences are, though.It would have been difficult to mention all the details in email
(including examples and corner scenarios) and thus we thought that it
will be better to document everything in wiki page for the time being.
We can keep on discussing the design and all the scenarios on need
basis (before implementation phase of that part) and thus eventually
everything will come in email on hackers. With out first patch, we
plan to provide everything in a README as well.
The challenge with having this on wiki is that it's unlikely people will
notice any changes made to the wiki.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 5/28/24 11:17, Nisha Moond wrote:
On Mon, May 27, 2024 at 11:19 AM shveta malik <shveta.malik@gmail.com> wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:...
I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.Regarding "TOAST column" for deleted row cases, we may need to dig
more. Thanks for bringing this case. Let me analyze more here.I tested a simple case with a table with one TOAST column and found
that when a tuple with a TOAST column is deleted, both the tuple and
corresponding pg_toast entries are marked as ‘deleted’ (dead) but not
removed immediately. The main tuple and respective pg_toast entry are
permanently deleted only during vacuum. First, the main table’s dead
tuples are vacuumed, followed by the secondary TOAST relation ones (if
available).
Please let us know if you have a specific scenario in mind where the
TOAST column data is deleted immediately upon ‘delete’ operation,
rather than during vacuum, which we are missing.
I'm pretty sure you can vacuum the TOAST table directly, which means
you'll end up with a deleted tuple with TOAST pointers, but with the
TOAST entries already gone.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 6/3/24 09:30, Amit Kapila wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 5/23/24 08:36, shveta malik wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.Why not to have some support for user-defined conflict resolution
methods, allowing to do more complex stuff (e.g. merging the rows in
some way, perhaps even with datatype-specific behavior)?The change will be converted to 'UPDATE' and applied if the decision
is in favor of applying remote change.It is important to have commit timestamp info available on subscriber
when latest_timestamp_wins or earliest_timestamp_wins method is chosen
as resolution method. Thus ‘track_commit_timestamp’ must be enabled
on subscriber, in absence of which, configuring the said
timestamp-based resolution methods will result in error.Note: If the user has chosen the latest or earliest_timestamp_wins,
and the remote and local timestamps are the same, then it will go by
system identifier. The change with a higher system identifier will
win. This will ensure that the same change is picked on all the nodes.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.One of the possible scenarios discussed at pgconf.dev with Tomas for
this was as follows:Say there are two publisher nodes PN1, PN2, and subscriber node SN3.
The logical replication is configured such that a subscription on SN3
has publications from both PN1 and PN2. For example, SN3 (sub) -> PN1,
PN2 (p1, p2)Now, on PN1, we have the following operations that update the same row:
T1
Update-1 on table t1 at LSN1 (1000) on time (200)T2
Update-2 on table t1 at LSN2 (2000) on time (100)Then in parallel, we have the following operation on node PN2 that
updates the same row as Update-1, and Update-2 on node PN1.T3
Update-3 on table t1 at LSN(1500) on time (150)By theory, we can have a different state on subscribers depending on
the order of updates arriving at SN3 which shouldn't happen. Say, the
order in which they reach SN3 is: Update-1, Update-2, Update-3 then
the final row we have is by Update-3 considering we have configured
last_update_wins as a conflict resolution method. Now, consider the
other order: Update-1, Update-3, Update-2, in this case, the final
row will be by Update-2 because when we try to apply Update-3, it will
generate a conflict and as per the resolution method
(last_update_wins) we need to retain Update-1.On further thinking, the operations on node-1 PN-1 as defined above
seem impossible because one of the Updates needs to wait for the other
to write a commit record. So the commits may happen with LSN1 < LSN2
but with T1 > T2 but they can't be on the same row due to locks. So,
the order of apply should still be consistent. Am, I missing
something?
Sorry, I should have read your message before responding a couple
minutes ago. I think you're right this exact example can't happen, due
to the dependency between transactions.
But as I wrote, I'm not quite convinced this means there are not other
issues with this way of resolving conflicts. It's more likely a more
complex scenario is required.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Fri, Jun 7, 2024 at 5:39 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
On Thu, Jun 6, 2024 at 5:16 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Here are more use cases of the "earliest_timestamp_wins" resolution method:
1) Applications where the record of first occurrence of an event is
important. For example, sensor based applications like earthquake
detection systems, capturing the first seismic wave's time is crucial.
2) Scheduling systems, like appointment booking, prioritize the
earliest request when handling concurrent ones.
3) In contexts where maintaining chronological order is important -
a) Social media platforms display comments ensuring that the
earliest ones are visible first.
b) Finance transaction processing systems rely on timestamps to
prioritize the processing of transactions, ensuring that the earliest
transaction is handled firstThanks for sharing examples. However, these scenarios would be handled by the application and not during replication. What we are discussing here is the timestamp when a row was updated/inserted/deleted (or rather when the transaction that updated row committed/became visible) and not a DML on column which is of type timestamp. Some implementations use a hidden timestamp column but that's different from a user column which captures timestamp of (say) an event. The conflict resolution will be based on the timestamp when that column's value was recorded in the database which may be different from the value of the column itself.
It depends on how these operations are performed. For example, the
appointment booking system could be prioritized via a transaction
updating a row with columns emp_name, emp_id, reserved, time_slot.
Now, if two employees at different geographical locations try to book
the calendar, the earlier transaction will win.
If we use the transaction commit timestamp as basis for resolution, a transaction where multiple rows conflict may end up with different rows affected by that transaction being resolved differently. Say three transactions T1, T2 and T3 on separate origins with timestamps t1, t2, and t3 respectively changed rows r1, r2 and r2, r3 and r1, r4 respectively. Changes to r1 and r2 will conflict. Let's say T2 and T3 are applied first and then T1 is applied. If t2 < t1 < t3, r1 will end up with version of T3 and r2 will end up with version of T1 after applying all the three transactions.
Are you telling the results based on latest_timestamp_wins? If so,
then it is correct. OTOH, if the user has configured
"earliest_timestamp_wins" resolution method, then we should end up
with a version of r1 from T1 because t1 < t3. Also, due to the same
reason, we should have version r2 from T2.
Would that introduce an inconsistency between r1 and r2?
As per my understanding, this shouldn't be an inconsistency. Won't it
be true even when the transactions are performed on a single node with
the same timing?
--
With Regards,
Amit Kapila.
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 5/27/24 07:48, shveta malik wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.Can you please explain a little bit more on this.
I was referring to the well established consistency models / isolation
levels, e.g. READ COMMITTED or SNAPSHOT ISOLATION. This determines what
guarantees the application developer can expect, what anomalies can
happen, etc.I don't think any such isolation level can be implemented with a simple
conflict resolution methods like last-update-wins etc. For example,
consider an active-active where both nodes doUPDATE accounts SET balance=balance+1000 WHERE id=1
This will inevitably lead to a conflict, and while the last-update-wins
resolves this "consistently" on both nodes (e.g. ending with the same
result), it's essentially a lost update.
The idea to solve such conflicts is using the delta apply technique
where the delta from both sides will be applied to the respective
columns. We do plan to target this as a separate patch. Now, if the
basic conflict resolution and delta apply both can't go in one
release, we shall document such cases clearly to avoid misuse of the
feature.
This is a very simplistic example of course, I recall there are various
more complex examples involving foreign keys, multi-table transactions,
constraints, etc. But in principle it's a manifestation of the same
inherent limitation of conflict detection and resolution etc.Similarly, I believe this affects not just active-active, but also the
case where one node aggregates data from multiple publishers. Maybe not
to the same extent / it might be fine for that use case,
I am not sure how much it is a problem for general logical replication
solution but we do intend to work on solving such problems in
step-wise manner. Trying to attempt everything in one patch doesn't
seem advisable to me.
but you said
the end goal is to use this for active-active. So I'm wondering what's
the plan, there.
I think at this stage we are not ready for active-active because
leaving aside this feature we need many other features like
replication of all commands/objects (DDL replication, replicate large
objects, etc.), Global sequences, some sort of global two_phase
transaction management for data consistency, etc. So, it would be
better to consider logical replication cases intending to extend it
for active-active when we have other required pieces.
If I'm writing an application for active-active using this conflict
handling, what assumptions can I make? Will Can I just do stuff as if on
a single node, or do I need to be super conscious about the zillion ways
things can misbehave in a distributed system?My personal opinion is that the closer this will be to the regular
consistency levels, the better. If past experience taught me anything,
it's very hard to predict how distributed systems with eventual
consistency behave, and even harder to actually test the application in
such environment.
I don't think in any way this can enable users to start writing
applications for active-active workloads. For something like what you
are saying, we probably need a global transaction manager (or a global
two_pc) as well to allow transactions to behave as they are on
single-node or achieve similar consistency levels. With such
transaction management, we can allow transactions to commit on a node
only when it doesn't lead to a conflict on the peer node.
In any case, if there are any differences compared to the usual
behavior, it needs to be very clearly explained in the docs.
I agree that docs should be clear about the cases that this can and
can't support.
How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.Are you pointing to the issue where a session/txn has taken
'xactStopTimestamp' timestamp earlier but is delayed to insert record
in XLOG, while another session/txn which has taken timestamp slightly
later succeeded to insert the record IN XLOG sooner than the session1,
making LSN and Timestamps out of sync? Going by this scenario, the
commit-timestamp may not be reflective of actual commits and thus
timestamp-based resolvers may take wrong decisions. Or do you mean
something else?If this is the problem you are referring to, then I think this needs a
fix at the publisher side. Let me think more about it . Kindly let me
know if you have ideas on how to tackle it.Yes, this is the issue I'm talking about. We're acquiring the timestamp
when not holding the lock to reserve space in WAL, so the LSN and the
commit LSN may not actually correlate.Consider this example I discussed with Amit last week:
node A:
XACT1: UPDATE t SET v = 1; LSN1 / T1
XACT2: UPDATE t SET v = 2; LSN2 / T2
node B
XACT3: UPDATE t SET v = 3; LSN3 / T3
And assume LSN1 < LSN2, T1 > T2 (i.e. the commit timestamp inversion),
and T2 < T3 < T1. Now consider that the messages may arrive in different
orders, due to async replication. Unfortunately, this would lead to
different results of the conflict resolution:XACT1 - XACT2 - XACT3 => v=3 (T3 wins)
XACT3 - XACT1 - XACT2 => v=2 (T2 wins)
Now, I realize there's a flaw in this example - the (T1 > T2) inversion
can't actually happen, because these transactions have a dependency, and
thus won't commit concurrently. XACT1 will complete the commit, because
XACT2 starts to commit. And with monotonic clock (which is a requirement
for any timestamp-based resolution), that should guarantee (T1 < T2).However, I doubt this is sufficient to declare victory. It's more likely
that there still are problems, but the examples are likely more complex
(changes to multiple tables, etc.).
Fair enough, I think we need to analyze this more to find actual
problems or in some way try to prove that there is no problem.
I vaguely remember there were more issues with timestamp inversion, but
those might have been related to parallel apply etc.
Okay, so considering there are problems due to timestamp inversion, I
think the solution to that problem would probably be somehow
generating commit LSN and timestamp in order. I don't have a solution
at this stage but will think more both on the actual problem and
solution. In the meantime, if you get a chance to refer to the place
where you have seen such a problem please try to share the same with
us. It would be helpful.
--
With Regards,
Amit Kapila.
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.
Doesn't the above example indicate that 'update_deleted' should also
be considered a necessary conflict type? Please see the possibilities
of conflicts in all three cases:
The "correct" order of receiving messages on node C (as suggested
above) is T1-T3-T2 (case1)
----------
T1 will insert the row.
T3 will have update_differ conflict; latest_timestamp wins or apply
will apply it. earliest_timestamp_wins or skip will skip it.
T2 will delete the row (irrespective of whether the update happened or not).
End Result: No Data.
T1-T2-T3
----------
T1 will insert the row.
T2 will delete the row.
T3 will have conflict update_deleted. If it is 'update_deleted', the
chances are that the resolver set here is to 'skip' (default is also
'skip' in this case).
If vacuum has deleted that row (or if we don't support
'update_deleted' conflict), it will be 'update_missing' conflict. In
that case, the user may end up inserting the row if resolver chosen is
in favor of apply (which seems an obvious choice for 'update_missing'
conflict; default is also 'apply_or_skip').
End result:
Row inserted with 'update_missing'.
Row correctly skipped with 'update_deleted' (assuming the obvious
choice seems to be 'skip' for update_deleted case).
So it seems that with 'update_deleted' conflict, there are higher
chances of opting for right decision here (which is to discard the
update), as 'update_deleted' conveys correct info to the user. The
'update_missing' OTOH does not convey correct info and user may end up
inserting the data by choosing apply favoring resolvers for
'update_missing'. Again, we get benefit of 'update_deleted' for
*recently* deleted rows only.
T3-T1-T2
----------
T3 may end up inserting the record if the resolver is in favor of
'apply' and all the columns are received from remote.
T1 will have' insert_exists' conflict and thus may either overwrite
'updated' values or may leave the data as is (based on whether
resolver is in favor of apply or not)
T2 will end up deleting it.
End Result: No Data.
I feel for second case (and similar cases), 'update_deleted' serves a
better conflict type.
thanks
Shveta
On Fri, Jun 7, 2024 at 6:10 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.Regarding "TOAST column" for deleted row cases, we may need to dig
more. Thanks for bringing this case. Let me analyze more here.I tested a simple case with a table with one TOAST column and found
that when a tuple with a TOAST column is deleted, both the tuple and
corresponding pg_toast entries are marked as ‘deleted’ (dead) but not
removed immediately. The main tuple and respective pg_toast entry are
permanently deleted only during vacuum. First, the main table’s dead
tuples are vacuumed, followed by the secondary TOAST relation ones (if
available).
Please let us know if you have a specific scenario in mind where the
TOAST column data is deleted immediately upon ‘delete’ operation,
rather than during vacuum, which we are missing.I'm pretty sure you can vacuum the TOAST table directly, which means
you'll end up with a deleted tuple with TOAST pointers, but with the
TOAST entries already gone.
It is true that for a deleted row, its toast entries can be vacuumed
earlier than the original/parent row, but we do not need to be
concerned about that to raise 'update_deleted'. To raise an
'update_deleted' conflict, it is sufficient to know that the row has
been deleted and not yet vacuumed, regardless of the presence or
absence of its toast entries. Once this is determined, we need to
build the tuple from remote data and apply it (provided resolver is
such that). If the tuple cannot be fully constructed from the remote
data, the apply operation will either be skipped or an error will be
raised, depending on whether the user has chosen the apply_or_skip or
apply_or_error option.
In cases where the table has toast columns but the remote data does
not include the toast-column entry (when the toast column is
unmodified and not part of the replica identity), the resolution for
'update_deleted' will be no worse than for 'update_missing'. That is,
for both the cases, we can not construct full tuple and thus the
operation either needs to be skipped or error needs to be raised.
thanks
Shveta
On 6/10/24 10:54, Amit Kapila wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 5/27/24 07:48, shveta malik wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.Can you please explain a little bit more on this.
I was referring to the well established consistency models / isolation
levels, e.g. READ COMMITTED or SNAPSHOT ISOLATION. This determines what
guarantees the application developer can expect, what anomalies can
happen, etc.I don't think any such isolation level can be implemented with a simple
conflict resolution methods like last-update-wins etc. For example,
consider an active-active where both nodes doUPDATE accounts SET balance=balance+1000 WHERE id=1
This will inevitably lead to a conflict, and while the last-update-wins
resolves this "consistently" on both nodes (e.g. ending with the same
result), it's essentially a lost update.The idea to solve such conflicts is using the delta apply technique
where the delta from both sides will be applied to the respective
columns. We do plan to target this as a separate patch. Now, if the
basic conflict resolution and delta apply both can't go in one
release, we shall document such cases clearly to avoid misuse of the
feature.
Perhaps, but it's not like having delta conflict resolution (or even
CRDT as a more generic variant) would lead to a regular consistency
model in a distributed system. At least I don't think it can achieve
that, because of the asynchronicity.
Consider a table with "CHECK (amount < 1000)" constraint, and an update
that sets (amount = amount + 900) on two nodes. AFAIK there's no way to
reconcile this using delta (or any other other) conflict resolution.
Which does not mean we should not have some form of conflict resolution,
as long as we know what the goal is. I simply don't want to spend time
working on this, add a lot of complex code, and then realize it doesn't
give us a consistency model that makes sense.
Which leads me back to my original question - what is the consistency
model this you expect to get from this (possibly when combined with some
other pieces?)?
This is a very simplistic example of course, I recall there are various
more complex examples involving foreign keys, multi-table transactions,
constraints, etc. But in principle it's a manifestation of the same
inherent limitation of conflict detection and resolution etc.Similarly, I believe this affects not just active-active, but also the
case where one node aggregates data from multiple publishers. Maybe not
to the same extent / it might be fine for that use case,I am not sure how much it is a problem for general logical replication
solution but we do intend to work on solving such problems in
step-wise manner. Trying to attempt everything in one patch doesn't
seem advisable to me.
I didn't say it needs to be done in one patch. I asked for someone to
explain what is the goal - consistency model observed by the users.
but you said
the end goal is to use this for active-active. So I'm wondering what's
the plan, there.I think at this stage we are not ready for active-active because
leaving aside this feature we need many other features like
replication of all commands/objects (DDL replication, replicate large
objects, etc.), Global sequences, some sort of global two_phase
transaction management for data consistency, etc. So, it would be
better to consider logical replication cases intending to extend it
for active-active when we have other required pieces.
We're not ready for active-active, sure. And I'm not saying a conflict
resolution would make us ready. The question is what consistency model
we'd like to get from the active-active, and whether conflict resolution
can get us there ...
As for the other missing bits (DDL replication, large objects, global
sequences), I think those are somewhat independent of the question I'm
asking. And some of the stuff is also somewhat optional - for example I
think it'd be fine to not support large objects or global sequences.
If I'm writing an application for active-active using this conflict
handling, what assumptions can I make? Will Can I just do stuff as if on
a single node, or do I need to be super conscious about the zillion ways
things can misbehave in a distributed system?My personal opinion is that the closer this will be to the regular
consistency levels, the better. If past experience taught me anything,
it's very hard to predict how distributed systems with eventual
consistency behave, and even harder to actually test the application in
such environment.I don't think in any way this can enable users to start writing
applications for active-active workloads. For something like what you
are saying, we probably need a global transaction manager (or a global
two_pc) as well to allow transactions to behave as they are on
single-node or achieve similar consistency levels. With such
transaction management, we can allow transactions to commit on a node
only when it doesn't lead to a conflict on the peer node.
But the wiki linked in the first message says:
CDR is an important and necessary feature for active-active
replication.
But if I understand your response, you're saying active-active should
probably use global transaction manager etc. which would prevent
conflicts - but seems to make CDR unnecessary. Or do I understand it wrong?
FWIW I don't think we'd need global components, there are ways to do
distributed snapshots using timestamps (for example), which would give
us snapshot isolation.
In any case, if there are any differences compared to the usual
behavior, it needs to be very clearly explained in the docs.I agree that docs should be clear about the cases that this can and
can't support.How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.Are you pointing to the issue where a session/txn has taken
'xactStopTimestamp' timestamp earlier but is delayed to insert record
in XLOG, while another session/txn which has taken timestamp slightly
later succeeded to insert the record IN XLOG sooner than the session1,
making LSN and Timestamps out of sync? Going by this scenario, the
commit-timestamp may not be reflective of actual commits and thus
timestamp-based resolvers may take wrong decisions. Or do you mean
something else?If this is the problem you are referring to, then I think this needs a
fix at the publisher side. Let me think more about it . Kindly let me
know if you have ideas on how to tackle it.Yes, this is the issue I'm talking about. We're acquiring the timestamp
when not holding the lock to reserve space in WAL, so the LSN and the
commit LSN may not actually correlate.Consider this example I discussed with Amit last week:
node A:
XACT1: UPDATE t SET v = 1; LSN1 / T1
XACT2: UPDATE t SET v = 2; LSN2 / T2
node B
XACT3: UPDATE t SET v = 3; LSN3 / T3
And assume LSN1 < LSN2, T1 > T2 (i.e. the commit timestamp inversion),
and T2 < T3 < T1. Now consider that the messages may arrive in different
orders, due to async replication. Unfortunately, this would lead to
different results of the conflict resolution:XACT1 - XACT2 - XACT3 => v=3 (T3 wins)
XACT3 - XACT1 - XACT2 => v=2 (T2 wins)
Now, I realize there's a flaw in this example - the (T1 > T2) inversion
can't actually happen, because these transactions have a dependency, and
thus won't commit concurrently. XACT1 will complete the commit, because
XACT2 starts to commit. And with monotonic clock (which is a requirement
for any timestamp-based resolution), that should guarantee (T1 < T2).However, I doubt this is sufficient to declare victory. It's more likely
that there still are problems, but the examples are likely more complex
(changes to multiple tables, etc.).Fair enough, I think we need to analyze this more to find actual
problems or in some way try to prove that there is no problem.I vaguely remember there were more issues with timestamp inversion, but
those might have been related to parallel apply etc.Okay, so considering there are problems due to timestamp inversion, I
think the solution to that problem would probably be somehow
generating commit LSN and timestamp in order. I don't have a solution
at this stage but will think more both on the actual problem and
solution. In the meantime, if you get a chance to refer to the place
where you have seen such a problem please try to share the same with
us. It would be helpful.
I think the solution to this would be to acquire the timestamp while
reserving the space (because that happens in LSN order). The clock would
need to be monotonic (easy enough with CLOCK_MONOTONIC), but also cheap.
AFAIK this is the main problem why it's being done outside the critical
section, because gettimeofday() may be quite expensive. There's a
concept of hybrid clock, combining "time" and logical counter, which I
think might be useful independently of CDR ...
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 6/10/24 12:56, shveta malik wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.Doesn't the above example indicate that 'update_deleted' should also
be considered a necessary conflict type? Please see the possibilities
of conflicts in all three cases:The "correct" order of receiving messages on node C (as suggested
above) is T1-T3-T2 (case1)
----------
T1 will insert the row.
T3 will have update_differ conflict; latest_timestamp wins or apply
will apply it. earliest_timestamp_wins or skip will skip it.
T2 will delete the row (irrespective of whether the update happened or not).
End Result: No Data.T1-T2-T3
----------
T1 will insert the row.
T2 will delete the row.
T3 will have conflict update_deleted. If it is 'update_deleted', the
chances are that the resolver set here is to 'skip' (default is also
'skip' in this case).If vacuum has deleted that row (or if we don't support
'update_deleted' conflict), it will be 'update_missing' conflict. In
that case, the user may end up inserting the row if resolver chosen is
in favor of apply (which seems an obvious choice for 'update_missing'
conflict; default is also 'apply_or_skip').End result:
Row inserted with 'update_missing'.
Row correctly skipped with 'update_deleted' (assuming the obvious
choice seems to be 'skip' for update_deleted case).So it seems that with 'update_deleted' conflict, there are higher
chances of opting for right decision here (which is to discard the
update), as 'update_deleted' conveys correct info to the user. The
'update_missing' OTOH does not convey correct info and user may end up
inserting the data by choosing apply favoring resolvers for
'update_missing'. Again, we get benefit of 'update_deleted' for
*recently* deleted rows only.T3-T1-T2
----------
T3 may end up inserting the record if the resolver is in favor of
'apply' and all the columns are received from remote.
T1 will have' insert_exists' conflict and thus may either overwrite
'updated' values or may leave the data as is (based on whether
resolver is in favor of apply or not)
T2 will end up deleting it.
End Result: No Data.I feel for second case (and similar cases), 'update_deleted' serves a
better conflict type.
True, but this is pretty much just a restatement of the example, right?
The point I was trying to make is that this hinges on the ability to
detect the correct conflict type. And if vacuum can swoop in and remove
the recently deleted tuples (which I believe can happen at any time,
right?), then that's not guaranteed, because we won't see the deleted
tuple anymore. Or am I missing something?
Also, can the resolver even convert the UPDATE into INSERT and proceed?
Maybe with REPLICA IDENTITY FULL? Otherwise the row might be incomplete,
missing required columns etc. In which case it'd have to wait for the
actual INSERT to arrive - which would work for actual update_missing,
where the row may be delayed due to network issues. But if that's a
mistake due to vacuum removing the deleted tuple, it'll wait forever.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Jun 10, 2024 at 5:24 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 6/10/24 12:56, shveta malik wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.Doesn't the above example indicate that 'update_deleted' should also
be considered a necessary conflict type? Please see the possibilities
of conflicts in all three cases:The "correct" order of receiving messages on node C (as suggested
above) is T1-T3-T2 (case1)
----------
T1 will insert the row.
T3 will have update_differ conflict; latest_timestamp wins or apply
will apply it. earliest_timestamp_wins or skip will skip it.
T2 will delete the row (irrespective of whether the update happened or not).
End Result: No Data.T1-T2-T3
----------
T1 will insert the row.
T2 will delete the row.
T3 will have conflict update_deleted. If it is 'update_deleted', the
chances are that the resolver set here is to 'skip' (default is also
'skip' in this case).If vacuum has deleted that row (or if we don't support
'update_deleted' conflict), it will be 'update_missing' conflict. In
that case, the user may end up inserting the row if resolver chosen is
in favor of apply (which seems an obvious choice for 'update_missing'
conflict; default is also 'apply_or_skip').End result:
Row inserted with 'update_missing'.
Row correctly skipped with 'update_deleted' (assuming the obvious
choice seems to be 'skip' for update_deleted case).So it seems that with 'update_deleted' conflict, there are higher
chances of opting for right decision here (which is to discard the
update), as 'update_deleted' conveys correct info to the user. The
'update_missing' OTOH does not convey correct info and user may end up
inserting the data by choosing apply favoring resolvers for
'update_missing'. Again, we get benefit of 'update_deleted' for
*recently* deleted rows only.T3-T1-T2
----------
T3 may end up inserting the record if the resolver is in favor of
'apply' and all the columns are received from remote.
T1 will have' insert_exists' conflict and thus may either overwrite
'updated' values or may leave the data as is (based on whether
resolver is in favor of apply or not)
T2 will end up deleting it.
End Result: No Data.I feel for second case (and similar cases), 'update_deleted' serves a
better conflict type.True, but this is pretty much just a restatement of the example, right?
The point I was trying to make is that this hinges on the ability to
detect the correct conflict type. And if vacuum can swoop in and remove
the recently deleted tuples (which I believe can happen at any time,
right?), then that's not guaranteed, because we won't see the deleted
tuple anymore.
Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.
Also, can the resolver even convert the UPDATE into INSERT and proceed?
Maybe with REPLICA IDENTITY FULL?
Yes, it can, as long as the row doesn't contain toasted data. Without
toasted data, the new tuple is fully logged. However, if the row does
contain toasted data, the new tuple won't log it completely. In such a
case, REPLICA IDENTITY FULL becomes a requirement to ensure we have
all the data necessary to create the row on the target side. In
absence of RI full and with row lacking toasted data, the operation
will be skipped or error will be raised.
Otherwise the row might be incomplete,
missing required columns etc. In which case it'd have to wait for the
actual INSERT to arrive - which would work for actual update_missing,
where the row may be delayed due to network issues. But if that's a
mistake due to vacuum removing the deleted tuple, it'll wait forever.
Even in case of 'update_missing', we do not intend to wait for 'actual
insert' to arrive, as it is not guaranteed if the 'insert' will arrive
or not. And thus we plan to skip or error out (based on user's
configuration) if a complete row can not be created for insertion.
thanks
Shveta
On Sat, Jun 8, 2024 at 3:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Jun 7, 2024 at 5:39 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:On Thu, Jun 6, 2024 at 5:16 PM Nisha Moond <nisha.moond412@gmail.com>
wrote:
Here are more use cases of the "earliest_timestamp_wins" resolution
method:
1) Applications where the record of first occurrence of an event is
important. For example, sensor based applications like earthquake
detection systems, capturing the first seismic wave's time is crucial.
2) Scheduling systems, like appointment booking, prioritize the
earliest request when handling concurrent ones.
3) In contexts where maintaining chronological order is important -
a) Social media platforms display comments ensuring that the
earliest ones are visible first.
b) Finance transaction processing systems rely on timestamps to
prioritize the processing of transactions, ensuring that the earliest
transaction is handled firstThanks for sharing examples. However, these scenarios would be handled
by the application and not during replication. What we are discussing here
is the timestamp when a row was updated/inserted/deleted (or rather when
the transaction that updated row committed/became visible) and not a DML on
column which is of type timestamp. Some implementations use a hidden
timestamp column but that's different from a user column which captures
timestamp of (say) an event. The conflict resolution will be based on the
timestamp when that column's value was recorded in the database which may
be different from the value of the column itself.It depends on how these operations are performed. For example, the
appointment booking system could be prioritized via a transaction
updating a row with columns emp_name, emp_id, reserved, time_slot.
Now, if two employees at different geographical locations try to book
the calendar, the earlier transaction will win.
I doubt that it would be that simple. The application will have to
intervene and tell one of the employees that their reservation has failed.
It looks natural that the first one to reserve the room should get the
reservation, but implementing that is more complex than resolving a
conflict in the database. In fact, mostly it will be handled outside
database.
If we use the transaction commit timestamp as basis for resolution, a
transaction where multiple rows conflict may end up with different rows
affected by that transaction being resolved differently. Say three
transactions T1, T2 and T3 on separate origins with timestamps t1, t2, and
t3 respectively changed rows r1, r2 and r2, r3 and r1, r4 respectively.
Changes to r1 and r2 will conflict. Let's say T2 and T3 are applied first
and then T1 is applied. If t2 < t1 < t3, r1 will end up with version of T3
and r2 will end up with version of T1 after applying all the three
transactions.Are you telling the results based on latest_timestamp_wins? If so,
then it is correct. OTOH, if the user has configured
"earliest_timestamp_wins" resolution method, then we should end up
with a version of r1 from T1 because t1 < t3. Also, due to the same
reason, we should have version r2 from T2.Would that introduce an inconsistency between r1 and r2?
As per my understanding, this shouldn't be an inconsistency. Won't it
be true even when the transactions are performed on a single node with
the same timing?
The inconsistency will arise irrespective of conflict resolution method. On
a single system effects of whichever transaction runs last will be visible
entirely. But in the example above the node where T1, T2, and T3 (from
*different*) origins) are applied, we might end up with a situation where
some changes from T1 are applied whereas some changes from T3 are applied.
--
Best Wishes,
Ashutosh Bapat
On 6/11/24 10:35, shveta malik wrote:
On Mon, Jun 10, 2024 at 5:24 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 6/10/24 12:56, shveta malik wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.Doesn't the above example indicate that 'update_deleted' should also
be considered a necessary conflict type? Please see the possibilities
of conflicts in all three cases:The "correct" order of receiving messages on node C (as suggested
above) is T1-T3-T2 (case1)
----------
T1 will insert the row.
T3 will have update_differ conflict; latest_timestamp wins or apply
will apply it. earliest_timestamp_wins or skip will skip it.
T2 will delete the row (irrespective of whether the update happened or not).
End Result: No Data.T1-T2-T3
----------
T1 will insert the row.
T2 will delete the row.
T3 will have conflict update_deleted. If it is 'update_deleted', the
chances are that the resolver set here is to 'skip' (default is also
'skip' in this case).If vacuum has deleted that row (or if we don't support
'update_deleted' conflict), it will be 'update_missing' conflict. In
that case, the user may end up inserting the row if resolver chosen is
in favor of apply (which seems an obvious choice for 'update_missing'
conflict; default is also 'apply_or_skip').End result:
Row inserted with 'update_missing'.
Row correctly skipped with 'update_deleted' (assuming the obvious
choice seems to be 'skip' for update_deleted case).So it seems that with 'update_deleted' conflict, there are higher
chances of opting for right decision here (which is to discard the
update), as 'update_deleted' conveys correct info to the user. The
'update_missing' OTOH does not convey correct info and user may end up
inserting the data by choosing apply favoring resolvers for
'update_missing'. Again, we get benefit of 'update_deleted' for
*recently* deleted rows only.T3-T1-T2
----------
T3 may end up inserting the record if the resolver is in favor of
'apply' and all the columns are received from remote.
T1 will have' insert_exists' conflict and thus may either overwrite
'updated' values or may leave the data as is (based on whether
resolver is in favor of apply or not)
T2 will end up deleting it.
End Result: No Data.I feel for second case (and similar cases), 'update_deleted' serves a
better conflict type.True, but this is pretty much just a restatement of the example, right?
The point I was trying to make is that this hinges on the ability to
detect the correct conflict type. And if vacuum can swoop in and remove
the recently deleted tuples (which I believe can happen at any time,
right?), then that's not guaranteed, because we won't see the deleted
tuple anymore.Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.
I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.
I'm not sure dropping update_deleted entirely would be a good idea,
though. It pretty much guarantees making the wrong decision at least
sometimes. But at least it's predictable and users are more likely to
notice that (compared to update_delete working on well-behaving systems,
and then failing when a node starts lagging or something).
That's my opinion, though, and I don't intend to stay in the way. But I
think the solution is not that difficult - something needs to prevent
cleanup of recently dead tuples (until the "relevant" changes are
received and applied from other nodes). I don't know if that could be
done based on information we have for subscriptions, or if we need
something new.
Also, can the resolver even convert the UPDATE into INSERT and proceed?
Maybe with REPLICA IDENTITY FULL?Yes, it can, as long as the row doesn't contain toasted data. Without
toasted data, the new tuple is fully logged. However, if the row does
contain toasted data, the new tuple won't log it completely. In such a
case, REPLICA IDENTITY FULL becomes a requirement to ensure we have
all the data necessary to create the row on the target side. In
absence of RI full and with row lacking toasted data, the operation
will be skipped or error will be raised.Otherwise the row might be incomplete,
missing required columns etc. In which case it'd have to wait for the
actual INSERT to arrive - which would work for actual update_missing,
where the row may be delayed due to network issues. But if that's a
mistake due to vacuum removing the deleted tuple, it'll wait forever.Even in case of 'update_missing', we do not intend to wait for 'actual
insert' to arrive, as it is not guaranteed if the 'insert' will arrive
or not. And thus we plan to skip or error out (based on user's
configuration) if a complete row can not be created for insertion.
If the UPDATE contains all the columns and can be turned into an INSERT,
then that seems reasonable. But I don't see how skipping it could work
in general (except for some very simple / specific use cases). I'm not
sure if you suggest to skip just the one UPDATE or transaction as a
whole, but it seems to me either of those options could easily lead to
all kinds of inconsistencies and user confusion.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.
When working with a distributed system, we must accept some form of
eventual consistency model. However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.
In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.
A common method to handle such cases is using vector clocks for
conflict resolution. "Vector clocks" allow us to track the causal
relationships between changes across nodes, ensuring that we can
correctly order events and resolve conflicts in a manner that respects
the "happens-before" relationship. This method helps maintain
consistency and predictability in the system despite issues like clock
skew.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Mon, Jun 10, 2024 at 5:12 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 6/10/24 10:54, Amit Kapila wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 5/27/24 07:48, shveta malik wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Which architecture are you aiming for? Here you talk about multiple
providers, but the wiki page mentions active-active. I'm not sure how
much this matters, but it might.Currently, we are working for multi providers case but ideally it
should work for active-active also. During further discussion and
implementation phase, if we find that, there are cases which will not
work in straight-forward way for active-active, then our primary focus
will remain to first implement it for multiple providers architecture.Also, what kind of consistency you expect from this? Because none of
these simple conflict resolution methods can give you the regular
consistency models we're used to, AFAICS.Can you please explain a little bit more on this.
I was referring to the well established consistency models / isolation
levels, e.g. READ COMMITTED or SNAPSHOT ISOLATION. This determines what
guarantees the application developer can expect, what anomalies can
happen, etc.I don't think any such isolation level can be implemented with a simple
conflict resolution methods like last-update-wins etc. For example,
consider an active-active where both nodes doUPDATE accounts SET balance=balance+1000 WHERE id=1
This will inevitably lead to a conflict, and while the last-update-wins
resolves this "consistently" on both nodes (e.g. ending with the same
result), it's essentially a lost update.The idea to solve such conflicts is using the delta apply technique
where the delta from both sides will be applied to the respective
columns. We do plan to target this as a separate patch. Now, if the
basic conflict resolution and delta apply both can't go in one
release, we shall document such cases clearly to avoid misuse of the
feature.Perhaps, but it's not like having delta conflict resolution (or even
CRDT as a more generic variant) would lead to a regular consistency
model in a distributed system. At least I don't think it can achieve
that, because of the asynchronicity.Consider a table with "CHECK (amount < 1000)" constraint, and an update
that sets (amount = amount + 900) on two nodes. AFAIK there's no way to
reconcile this using delta (or any other other) conflict resolution.
Right, in such a case an error will be generated and I agree that we
can't always reconcile the updates on different nodes and some data
loss is unavoidable with or without conflict resolution.
Which does not mean we should not have some form of conflict resolution,
as long as we know what the goal is. I simply don't want to spend time
working on this, add a lot of complex code, and then realize it doesn't
give us a consistency model that makes sense.Which leads me back to my original question - what is the consistency
model this you expect to get from this (possibly when combined with some
other pieces?)?
I don't think this feature per se (or some additional features like
delta apply) can help with improving/changing the consistency model
our current logical replication module provides (which as per my
understanding is an eventual consistency model). This feature will
help with reducing the number of cases where manual intervention is
required with configurable way to resolve conflicts. For example, for
primary key violation ERRORs, or when we intentionally overwrite the
data even when there is conflicting data present from different
origin, or for cases we simply skip the remote data when there is a
conflict in the local node.
To achieve consistent reads on all nodes we either need a distributed
transaction using a two-phase commit with some sort of quorum
protocol, or a sharded database with multiple primaries each
responsible for a unique partition of the data, or some other way. The
current proposal doesn't intend to implement any of those.
--
With Regards,
Amit Kapila.
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 6/11/24 10:35, shveta malik wrote:
On Mon, Jun 10, 2024 at 5:24 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:On 6/10/24 12:56, shveta malik wrote:
On Fri, Jun 7, 2024 at 6:08 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:UPDATE
================Conflict Detection Method:
--------------------------------
Origin conflict detection: The ‘origin’ info is used to detect
conflict which can be obtained from commit-timestamp generated for
incoming txn at the source node. To compare remote’s origin with the
local’s origin, we must have origin information for local txns as well
which can be obtained from commit-timestamp after enabling
‘track_commit_timestamp’ locally.
The one drawback here is the ‘origin’ information cannot be obtained
once the row is frozen and the commit-timestamp info is removed by
vacuum. For a frozen row, conflicts cannot be raised, and thus the
incoming changes will be applied in all the cases.Conflict Types:
----------------
a) update_differ: The origin of an incoming update's key row differs
from the local row i.e.; the row has already been updated locally or
by different nodes.
b) update_missing: The row with the same value as that incoming
update's key does not exist. Remote is trying to update a row which
does not exist locally.
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I don't understand the why should update_missing or update_deleted be
different, especially considering it's not detected reliably. And also
that even if we happen to find the row the associated TOAST data may
have already been removed. So why would this matter?Here, we are trying to tackle the case where the row is 'recently'
deleted i.e. concurrent UPDATE and DELETE on pub and sub. User may
want to opt for a different resolution in such a case as against the
one where the corresponding row was not even present in the first
place. The case where the row was deleted long back may not fall into
this category as there are higher chances that they have been removed
by vacuum and can be considered equivalent to the update_ missing
case.My point is that if we can't detect the difference reliably, it's not
very useful. Consider this example:Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
The "correct" order of received messages on a third node is T1-T3-T2.
But we may also see T1-T2-T3 and T3-T1-T2, e.g. due to network issues
and so on. For T1-T2-T3 the right decision is to discard the update,
while for T3-T1-T2 it's to either wait for the INSERT or wait for the
insert to arrive.But if we misdetect the situation, we either end up with a row that
shouldn't be there, or losing an update.Doesn't the above example indicate that 'update_deleted' should also
be considered a necessary conflict type? Please see the possibilities
of conflicts in all three cases:The "correct" order of receiving messages on node C (as suggested
above) is T1-T3-T2 (case1)
----------
T1 will insert the row.
T3 will have update_differ conflict; latest_timestamp wins or apply
will apply it. earliest_timestamp_wins or skip will skip it.
T2 will delete the row (irrespective of whether the update happened or not).
End Result: No Data.T1-T2-T3
----------
T1 will insert the row.
T2 will delete the row.
T3 will have conflict update_deleted. If it is 'update_deleted', the
chances are that the resolver set here is to 'skip' (default is also
'skip' in this case).If vacuum has deleted that row (or if we don't support
'update_deleted' conflict), it will be 'update_missing' conflict. In
that case, the user may end up inserting the row if resolver chosen is
in favor of apply (which seems an obvious choice for 'update_missing'
conflict; default is also 'apply_or_skip').End result:
Row inserted with 'update_missing'.
Row correctly skipped with 'update_deleted' (assuming the obvious
choice seems to be 'skip' for update_deleted case).So it seems that with 'update_deleted' conflict, there are higher
chances of opting for right decision here (which is to discard the
update), as 'update_deleted' conveys correct info to the user. The
'update_missing' OTOH does not convey correct info and user may end up
inserting the data by choosing apply favoring resolvers for
'update_missing'. Again, we get benefit of 'update_deleted' for
*recently* deleted rows only.T3-T1-T2
----------
T3 may end up inserting the record if the resolver is in favor of
'apply' and all the columns are received from remote.
T1 will have' insert_exists' conflict and thus may either overwrite
'updated' values or may leave the data as is (based on whether
resolver is in favor of apply or not)
T2 will end up deleting it.
End Result: No Data.I feel for second case (and similar cases), 'update_deleted' serves a
better conflict type.True, but this is pretty much just a restatement of the example, right?
The point I was trying to make is that this hinges on the ability to
detect the correct conflict type. And if vacuum can swoop in and remove
the recently deleted tuples (which I believe can happen at any time,
right?), then that's not guaranteed, because we won't see the deleted
tuple anymore.Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.I'm not sure dropping update_deleted entirely would be a good idea,
though. It pretty much guarantees making the wrong decision at least
sometimes. But at least it's predictable and users are more likely to
notice that (compared to update_delete working on well-behaving systems,
and then failing when a node starts lagging or something).That's my opinion, though, and I don't intend to stay in the way. But I
think the solution is not that difficult - something needs to prevent
cleanup of recently dead tuples (until the "relevant" changes are
received and applied from other nodes). I don't know if that could be
done based on information we have for subscriptions, or if we need
something new.
I agree that without update_deleted, there are higher chances of
making incorrect decisions in some cases. But not sure if relying on
delaying vacuum from removing such rows is a full proof plan. We
cannot predict if or when "relevant" changes will occur, so how long
should we delay the vacuum?
To address this problem, we may need a completely different approach.
One solution could be to store deleted rows in a separate table
(dead-rows-table) so we can consult that table for any deleted entries
at any time. Additionally, we would need methods to purge older data
from the dead-rows-table to prevent it from growing too large. This
would be a substantial project on its own, so we can aim to implement
some initial and simple conflict resolution methods first before
tackling this more complex solution.
Also, can the resolver even convert the UPDATE into INSERT and proceed?
Maybe with REPLICA IDENTITY FULL?Yes, it can, as long as the row doesn't contain toasted data. Without
toasted data, the new tuple is fully logged. However, if the row does
contain toasted data, the new tuple won't log it completely. In such a
case, REPLICA IDENTITY FULL becomes a requirement to ensure we have
all the data necessary to create the row on the target side. In
absence of RI full and with row lacking toasted data, the operation
will be skipped or error will be raised.Otherwise the row might be incomplete,
missing required columns etc. In which case it'd have to wait for the
actual INSERT to arrive - which would work for actual update_missing,
where the row may be delayed due to network issues. But if that's a
mistake due to vacuum removing the deleted tuple, it'll wait forever.Even in case of 'update_missing', we do not intend to wait for 'actual
insert' to arrive, as it is not guaranteed if the 'insert' will arrive
or not. And thus we plan to skip or error out (based on user's
configuration) if a complete row can not be created for insertion.If the UPDATE contains all the columns and can be turned into an INSERT,
then that seems reasonable. But I don't see how skipping it could work
in general (except for some very simple / specific use cases). I'm not
sure if you suggest to skip just the one UPDATE or transaction as a
whole, but it seems to me either of those options could easily lead to
all kinds of inconsistencies and user confusion.
Conflict resolution is row-based, meaning that whatever action we
choose (error or skip) applies to the specific change rather than the
entire transaction. I'm not sure if waiting indefinitely for an INSERT
to arrive is a good idea, as the node that triggered the INSERT might
be down for an extended period. At best, we could provide a
configuration parameter using which the apply worker waits for a
specified time period for the INSERT to arrive before either skipping
or throwing an error.
That said, even if we error out or skip and log without waiting for
the INSERT, we won't introduce any new inconsistencies. This is the
current behavior on pg-HEAD. But with options like apply_or_skip and
apply_or_error, we have a better chance of resolving conflicts by
constructing the complete row internally, without user's intervention.
There will still be some cases where we can't fully reconstruct the
row, but in those instances, the behavior won't be any worse than the
current pg-HEAD.
thanks
Shveta
On 6/12/24 06:32, Dilip Kumar wrote:
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model.
I'm not sure this is necessarily true. There are distributed databases
implementing (or aiming to) regular consistency models, without eventual
consistency. I'm not saying it's easy, but it shows eventual consistency
is not the only option.
However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.
Right. And this is precisely the focus or my questions - understanding
what behavior we aim for / expect in the end. Or said differently, what
anomalies / weird behavior would be considered expected.
Because that's important both for discussions about feasibility, etc.
And also for evaluation / reviews of the patch.
In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution. "Vector clocks" allow us to track the causal
relationships between changes across nodes, ensuring that we can
correctly order events and resolve conflicts in a manner that respects
the "happens-before" relationship. This method helps maintain
consistency and predictability in the system despite issues like clock
skew.
I'm familiar with the concept of vector clock (or logical clock in
general), but it's not clear to me how you plan to use this in the
context of the conflict handling. Can you elaborate/explain?
The way I see it, conflict handling is pretty tightly coupled with
regular commit timestamps and MVCC in general. How would you use vector
clock to change that?
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Wed, Jun 12, 2024 at 5:26 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model.I'm not sure this is necessarily true. There are distributed databases
implementing (or aiming to) regular consistency models, without eventual
consistency. I'm not saying it's easy, but it shows eventual consistency
is not the only option.
Right, that statement might not be completely accurate. Based on the
CAP theorem, when a network partition is unavoidable and availability
is expected, we often choose an eventual consistency model. However,
claiming that a fully consistent model is impossible in any
distributed system is incorrect, as it can be achieved using
mechanisms like Two-Phase Commit.
We must also accept that our PostgreSQL replication mechanism does not
guarantee a fully consistent model. Even with synchronous commit, it
only waits for the WAL to be replayed on the standby but does not
change the commit decision based on other nodes. This means, at most,
we can only guarantee "Read Your Write" consistency.
However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.Right. And this is precisely the focus or my questions - understanding
what behavior we aim for / expect in the end. Or said differently, what
anomalies / weird behavior would be considered expected.
Because that's important both for discussions about feasibility, etc.
And also for evaluation / reviews of the patch.
+1
In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution. "Vector clocks" allow us to track the causal
relationships between changes across nodes, ensuring that we can
correctly order events and resolve conflicts in a manner that respects
the "happens-before" relationship. This method helps maintain
consistency and predictability in the system despite issues like clock
skew.I'm familiar with the concept of vector clock (or logical clock in
general), but it's not clear to me how you plan to use this in the
context of the conflict handling. Can you elaborate/explain?The way I see it, conflict handling is pretty tightly coupled with
regular commit timestamps and MVCC in general. How would you use vector
clock to change that?
The issue with using commit timestamps is that, when multiple nodes
are involved, the commit timestamp won't accurately represent the
actual order of operations. There's no reliable way to determine the
perfect order of each operation happening on different nodes roughly
simultaneously unless we use some globally synchronized counter.
Generally, that order might not cause real issues unless one operation
is triggered by a previous operation, and relying solely on physical
timestamps would not detect that correctly.
We need some sort of logical counter, such as a vector clock, which
might be an independent counter on each node but can perfectly track
the causal order. For example, if NodeA observes an operation from
NodeB with a counter value of X, NodeA will adjust its counter to X+1.
This ensures that if NodeA has seen an operation from NodeB, its next
operation will appear to have occurred after NodeB's operation.
I admit that I haven't fully thought through how we could design such
version tracking in our logical replication protocol or how it would
fit into our system. However, my point is that we need to consider
something beyond commit timestamps to achieve reliable ordering.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 5, 2024 at 3:32 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
Hi,
This time at PGconf.dev[1], we had some discussions regarding this
project. The proposed approach is to split the work into two main
components. The first part focuses on conflict detection, which aims to
identify and report conflicts in logical replication. This feature will
enable users to monitor the unexpected conflicts that may occur. The
second part involves the actual conflict resolution. Here, we will provide
built-in resolutions for each conflict and allow user to choose which
resolution will be used for which conflict(as described in the initial
email of this thread).
I agree with this direction that we focus on conflict detection (and
logging) first and then develop conflict resolution on top of that.
Of course, we are open to alternative ideas and suggestions, and the
strategy above can be changed based on ongoing discussions and feedback
received.Here is the patch of the first part work, which adds a new parameter
detect_conflict for CREATE and ALTER subscription commands. This new
parameter will decide if subscription will go for conflict detection. By
default, conflict detection will be off for a subscription.When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.While there exist other conflict types in logical replication, such as an
incoming insert conflicting with an existing row due to a primary key or
unique index, these cases already result in constraint violation errors.
What does detect_conflict being true actually mean to users? I
understand that detect_conflict being true could introduce some
overhead to detect conflicts. But in terms of conflict detection, even
if detect_confict is false, we detect some conflicts such as
concurrent inserts with the same key. Once we introduce the complete
conflict detection feature, I'm not sure there is a case where a user
wants to detect only some particular types of conflict.
Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.
I feel that we should log all types of conflict in an uniform way. For
example, with detect_conflict being true, the update_differ conflict
is reported as "conflict %s detected on relation "%s"", whereas
concurrent inserts with the same key is reported as "duplicate key
value violates unique constraint "%s"", which could confuse users.
Ideally, I think that we log such conflict detection details (table
name, column name, conflict type, etc) to somewhere (e.g. a table or
server logs) so that the users can resolve them manually.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
On Thu, Jun 13, 2024 at 11:41 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Wed, Jun 5, 2024 at 3:32 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:Hi,
This time at PGconf.dev[1], we had some discussions regarding this
project. The proposed approach is to split the work into two main
components. The first part focuses on conflict detection, which aims to
identify and report conflicts in logical replication. This feature will
enable users to monitor the unexpected conflicts that may occur. The
second part involves the actual conflict resolution. Here, we will provide
built-in resolutions for each conflict and allow user to choose which
resolution will be used for which conflict(as described in the initial
email of this thread).I agree with this direction that we focus on conflict detection (and
logging) first and then develop conflict resolution on top of that.Of course, we are open to alternative ideas and suggestions, and the
strategy above can be changed based on ongoing discussions and feedback
received.Here is the patch of the first part work, which adds a new parameter
detect_conflict for CREATE and ALTER subscription commands. This new
parameter will decide if subscription will go for conflict detection. By
default, conflict detection will be off for a subscription.When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.While there exist other conflict types in logical replication, such as an
incoming insert conflicting with an existing row due to a primary key or
unique index, these cases already result in constraint violation errors.What does detect_conflict being true actually mean to users? I
understand that detect_conflict being true could introduce some
overhead to detect conflicts. But in terms of conflict detection, even
if detect_confict is false, we detect some conflicts such as
concurrent inserts with the same key. Once we introduce the complete
conflict detection feature, I'm not sure there is a case where a user
wants to detect only some particular types of conflict.
You are right that users would wish to detect the conflicts and
probably the extra effort would only be in the 'update_differ' case
where we need to consult committs module and that we will only do when
'track_commit_timestamp' is true. BTW, I think for Inserts with
primary/unique key violation, we should catch the ERROR and log it. If
we want to log the conflicts in a separate table then do we want to do
that in the catch block after getting pk violation or do an extra scan
before 'INSERT' to find the conflict? I think logging would need extra
cost especially if we want to LOG it in some table as you are
suggesting below that may need some option.
Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.I feel that we should log all types of conflict in an uniform way. For
example, with detect_conflict being true, the update_differ conflict
is reported as "conflict %s detected on relation "%s"", whereas
concurrent inserts with the same key is reported as "duplicate key
value violates unique constraint "%s"", which could confuse users.
Ideally, I think that we log such conflict detection details (table
name, column name, conflict type, etc) to somewhere (e.g. a table or
server logs) so that the users can resolve them manually.
It is good to think if there is a value in providing in
pg_conflicts_history kind of table which will have details of
conflicts that occurred and then we can extend it to have resolutions.
I feel we can anyway LOG the conflicts by default. Updating a separate
table with conflicts should be done by default or with a knob is a
point to consider.
--
With Regards,
Amit Kapila.
On 23.05.24 08:36, shveta malik wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestamp wins.
b) earliest_timestamp_wins: The change with earlier commit timestamp wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.
You might be aware of pglogical, which has similar conflict resolution
modes, but they appear to be spelled a bit different. It might be worth
reviewing this, so that we don't unnecessarily introduce differences.
https://github.com/2ndquadrant/pglogical?tab=readme-ov-file#conflicts
There might also be other inspiration to be found related to this in
pglogical documentation or code.
On 2024-Jun-07, Tomas Vondra wrote:
On 6/3/24 09:30, Amit Kapila wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.
But as I wrote, I'm not quite convinced this means there are not other
issues with this way of resolving conflicts. It's more likely a more
complex scenario is required.
Jan Wieck approached me during pgconf.dev to reproach me of this
problem. He also said he had some code to fix-up the commit TS
afterwards somehow, to make the sequence monotonically increasing.
Perhaps we should consider that, to avoid any problems caused by the
difference between LSN order and TS order. It might be quite
nightmarish to try to make the system work correctly without
reasonable constraints of that nature.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
On 6/13/24 7:28 AM, Amit Kapila wrote:
You are right that users would wish to detect the conflicts and
probably the extra effort would only be in the 'update_differ' case
where we need to consult committs module and that we will only do when
'track_commit_timestamp' is true. BTW, I think for Inserts with
primary/unique key violation, we should catch the ERROR and log it. If
we want to log the conflicts in a separate table then do we want to do
that in the catch block after getting pk violation or do an extra scan
before 'INSERT' to find the conflict? I think logging would need extra
cost especially if we want to LOG it in some table as you are
suggesting below that may need some option.Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.I feel that we should log all types of conflict in an uniform way. For
example, with detect_conflict being true, the update_differ conflict
is reported as "conflict %s detected on relation "%s"", whereas
concurrent inserts with the same key is reported as "duplicate key
value violates unique constraint "%s"", which could confuse users.
Ideally, I think that we log such conflict detection details (table
name, column name, conflict type, etc) to somewhere (e.g. a table or
server logs) so that the users can resolve them manually.It is good to think if there is a value in providing in
pg_conflicts_history kind of table which will have details of
conflicts that occurred and then we can extend it to have resolutions.
I feel we can anyway LOG the conflicts by default. Updating a separate
table with conflicts should be done by default or with a knob is a
point to consider.
+1 for logging conflicts uniformly, but I would +100 to exposing the log
in a way that's easy for the user to query (whether it's a system view
or a stat table). Arguably, I'd say that would be the most important
feature to come out of this effort.
Removing how conflicts are resolved, users want to know exactly what row
had a conflict, and users from other database systems that have dealt
with these issues will have tooling to be able to review and analyze if
a conflicts occur. This data is typically stored in a queryable table,
with data retained for N days. When you add in automatic conflict
resolution, users then want to have a record of how the conflict was
resolved, in case they need to manually update it.
Having this data in a table also gives the user opportunity to
understand conflict stats (e.g. conflict rates) and potentially identify
portions of the application and other parts of the system to optimize.
It also makes it easier to import to downstream systems that may perform
further analysis on conflict resolution, or alarm if a conflict rate
exceeds a certain threshold.
Thanks,
Jonathan
On Thu, May 23, 2024 at 2:37 AM shveta malik <shveta.malik@gmail.com> wrote:
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.
I think this design is categorically unacceptable. It amounts to
designing a feature that works except when it doesn't. I'm not exactly
sure how the proposal should be changed to avoid depending on the
timing of VACUUM, but I think it's absolutely not OK to depend on the
timing of VACUUm -- or, really, this is going to depend on the timing
of HOT-pruning, which will often happen almost instantly.
--
Robert Haas
EDB: http://www.enterprisedb.com
On Thu, Jun 13, 2024 at 7:00 PM Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2024-Jun-07, Tomas Vondra wrote:
On 6/3/24 09:30, Amit Kapila wrote:
On Sat, May 25, 2024 at 2:39 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
How is this going to deal with the fact that commit LSN and timestamps
may not correlate perfectly? That is, commits may happen with LSN1 <
LSN2 but with T1 > T2.But as I wrote, I'm not quite convinced this means there are not other
issues with this way of resolving conflicts. It's more likely a more
complex scenario is required.Jan Wieck approached me during pgconf.dev to reproach me of this
problem. He also said he had some code to fix-up the commit TS
afterwards somehow, to make the sequence monotonically increasing.
Perhaps we should consider that, to avoid any problems caused by the
difference between LSN order and TS order. It might be quite
nightmarish to try to make the system work correctly without
reasonable constraints of that nature.
I agree with this but the problem Jan was worried about was not
directly reproducible in what the PostgreSQL provides at least that is
what I understood then. We are also unable to think of a concrete
scenario where this is a problem but we are planning to spend more
time deriving a test to reproducible the problem.
--
With Regards,
Amit Kapila.
On Thu, Jun 13, 2024 at 11:18 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
On 6/13/24 7:28 AM, Amit Kapila wrote:
I feel that we should log all types of conflict in an uniform way. For
example, with detect_conflict being true, the update_differ conflict
is reported as "conflict %s detected on relation "%s"", whereas
concurrent inserts with the same key is reported as "duplicate key
value violates unique constraint "%s"", which could confuse users.
Ideally, I think that we log such conflict detection details (table
name, column name, conflict type, etc) to somewhere (e.g. a table or
server logs) so that the users can resolve them manually.It is good to think if there is a value in providing in
pg_conflicts_history kind of table which will have details of
conflicts that occurred and then we can extend it to have resolutions.
I feel we can anyway LOG the conflicts by default. Updating a separate
table with conflicts should be done by default or with a knob is a
point to consider.+1 for logging conflicts uniformly, but I would +100 to exposing the log
in a way that's easy for the user to query (whether it's a system view
or a stat table). Arguably, I'd say that would be the most important
feature to come out of this effort.
We can have both the system view and a stats table. The system view
could have some sort of cumulative stats data like how many times a
particular conflict had occurred and the table would provide detailed
information about the conflict. The one challenge I see in providing a
table is in its cleanup mechanism. We could prove a partitioned table
such that users can truncate/drop the not needed partitions or provide
a non-partitioned table where users can delete the old data in which
case they generate a work for auto vacuum.
Removing how conflicts are resolved, users want to know exactly what row
had a conflict, and users from other database systems that have dealt
with these issues will have tooling to be able to review and analyze if
a conflicts occur. This data is typically stored in a queryable table,
with data retained for N days. When you add in automatic conflict
resolution, users then want to have a record of how the conflict was
resolved, in case they need to manually update it.Having this data in a table also gives the user opportunity to
understand conflict stats (e.g. conflict rates) and potentially identify
portions of the application and other parts of the system to optimize.
It also makes it easier to import to downstream systems that may perform
further analysis on conflict resolution, or alarm if a conflict rate
exceeds a certain threshold.
Agreed those are good use cases to store conflict history.
--
With Regards,
Amit Kapila.
On Fri, Jun 14, 2024 at 12:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, May 23, 2024 at 2:37 AM shveta malik <shveta.malik@gmail.com> wrote:
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I think this design is categorically unacceptable. It amounts to
designing a feature that works except when it doesn't. I'm not exactly
sure how the proposal should be changed to avoid depending on the
timing of VACUUM, but I think it's absolutely not OK to depend on the
timing of VACUUm -- or, really, this is going to depend on the timing
of HOT-pruning, which will often happen almost instantly.
Agreed, above Tomas has speculated to have a way to avoid vacuum
cleaning dead tuples until the required changes are received and
applied. Shveta also mentioned another way to have deads-store (say a
table where deleted rows are stored for resolution) [1]/messages/by-id/CAJpy0uCov4JfZJeOvY0O21_gk9bcgNUDp4jf8+BbMp+EAv8cVQ@mail.gmail.com which is
similar to a technique used by some other databases. There is an
agreement to not rely on Vacuum to detect such a conflict but the
alternative is not clear. Currently, we are thinking to consider such
a conflict type as update_missing (The row with the same value as that
incoming update's key does not exist.). This is how the current HEAD
code behaves and LOGs the information (logical replication did not
find row to be updated ..).
[1]: /messages/by-id/CAJpy0uCov4JfZJeOvY0O21_gk9bcgNUDp4jf8+BbMp+EAv8cVQ@mail.gmail.com
--
With Regards,
Amit Kapila.
On Thursday, June 13, 2024 8:46 PM Peter Eisentraut <peter@eisentraut.org> wrote:
On 23.05.24 08:36, shveta malik wrote:
Conflict Resolution
----------------
a) latest_timestamp_wins: The change with later commit timestampwins.
b) earliest_timestamp_wins: The change with earlier commit timestamp
wins.
c) apply: Always apply the remote change.
d) skip: Remote change is skipped.
e) error: Error out on conflict. Replication is stopped, manual
action is needed.You might be aware of pglogical, which has similar conflict resolution modes,
but they appear to be spelled a bit different. It might be worth reviewing this,
so that we don't unnecessarily introduce differences.
Right. Some of the proposed resolution names are different from pglogical's
while the functionalities are the same. The following is the comparison with
pglogical:
latest_timestamp_wins(proposal) - last_update_wins(pglogical)
earliest_timestamp_wins(proposal) - first_update_wins(pglogical)
apply(proposal) - apply_remote(pglogical)
skip(proposal) - keep_local(pglogical)
I personally think the pglogical's names read more naturally. But others may
have different opinions on this.
https://github.com/2ndquadrant/pglogical?tab=readme-ov-file#conflicts
There might also be other inspiration to be found related to this in pglogical
documentation or code.
Another difference is that we allow users to specify different resolutions for
different conflicts, while pglogical allows specifying one resolution for all conflict.
I think the proposed approach offers more flexibility to users, which seems more
favorable to me.
Best Regards,
Hou zj
On 6/14/24 13:29, Amit Kapila wrote:
On Fri, Jun 14, 2024 at 12:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, May 23, 2024 at 2:37 AM shveta malik <shveta.malik@gmail.com> wrote:
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I think this design is categorically unacceptable. It amounts to
designing a feature that works except when it doesn't. I'm not exactly
sure how the proposal should be changed to avoid depending on the
timing of VACUUM, but I think it's absolutely not OK to depend on the
timing of VACUUm -- or, really, this is going to depend on the timing
of HOT-pruning, which will often happen almost instantly.Agreed, above Tomas has speculated to have a way to avoid vacuum
cleaning dead tuples until the required changes are received and
applied. Shveta also mentioned another way to have deads-store (say a
table where deleted rows are stored for resolution) [1] which is
similar to a technique used by some other databases. There is an
agreement to not rely on Vacuum to detect such a conflict but the
alternative is not clear.
I'm not sure I'd say I "speculated" about it - it's not like we don't
have ways to hold off cleanup for a while for various reasons
(long-running query, replication slot, hot-standby feedback, ...).
How exactly would that be implemented I don't know, but it seems like a
far simpler approach than inventing a new "dead store". It'd need logic
to let the vacuum to cleanup the stuff no longer needed, but so would
the dead store I think.
Currently, we are thinking to consider such
a conflict type as update_missing (The row with the same value as that
incoming update's key does not exist.). This is how the current HEAD
code behaves and LOGs the information (logical replication did not
find row to be updated ..).
I thought the agreement was we need both conflict types to get sensible
behavior, so proceeding with just the update_missing (mostly because we
don't know how to detect these conflicts reliably) seems like maybe not
be the right direction ...
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On 6/13/24 06:52, Dilip Kumar wrote:
On Wed, Jun 12, 2024 at 5:26 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model.I'm not sure this is necessarily true. There are distributed databases
implementing (or aiming to) regular consistency models, without eventual
consistency. I'm not saying it's easy, but it shows eventual consistency
is not the only option.Right, that statement might not be completely accurate. Based on the
CAP theorem, when a network partition is unavoidable and availability
is expected, we often choose an eventual consistency model. However,
claiming that a fully consistent model is impossible in any
distributed system is incorrect, as it can be achieved using
mechanisms like Two-Phase Commit.We must also accept that our PostgreSQL replication mechanism does not
guarantee a fully consistent model. Even with synchronous commit, it
only waits for the WAL to be replayed on the standby but does not
change the commit decision based on other nodes. This means, at most,
we can only guarantee "Read Your Write" consistency.
Perhaps, but even accepting eventual consistency does not absolve us
from actually defining what that means, ensuring it's sensible enough to
be practical/usable, and that it actually converges to a consistent
state (that's essentially the problem of the update conflict types,
because misdetecting it results in diverging results).
However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.Right. And this is precisely the focus or my questions - understanding
what behavior we aim for / expect in the end. Or said differently, what
anomalies / weird behavior would be considered expected.Because that's important both for discussions about feasibility, etc.
And also for evaluation / reviews of the patch.+1
In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution. "Vector clocks" allow us to track the causal
relationships between changes across nodes, ensuring that we can
correctly order events and resolve conflicts in a manner that respects
the "happens-before" relationship. This method helps maintain
consistency and predictability in the system despite issues like clock
skew.I'm familiar with the concept of vector clock (or logical clock in
general), but it's not clear to me how you plan to use this in the
context of the conflict handling. Can you elaborate/explain?The way I see it, conflict handling is pretty tightly coupled with
regular commit timestamps and MVCC in general. How would you use vector
clock to change that?The issue with using commit timestamps is that, when multiple nodes
are involved, the commit timestamp won't accurately represent the
actual order of operations. There's no reliable way to determine the
perfect order of each operation happening on different nodes roughly
simultaneously unless we use some globally synchronized counter.
Generally, that order might not cause real issues unless one operation
is triggered by a previous operation, and relying solely on physical
timestamps would not detect that correctly.
This whole conflict detection / resolution proposal is based on using
commit timestamps. Aren't you suggesting it can't really work with
commit timestamps?
FWIW there are ways to builds distributed consistency with timestamps,
as long as it's monotonic - e.g. clock-SI does that. It's not perfect,
but it shows it's possible.
However, I'm not we have to go there - it depends on what the goal is.
For a one-directional replication (multiple nodes replicating to the
same target) it might be sufficient if the conflict resolution is
"deterministic" (e.g. not dependent on the order in which the changes
are applied). I'm not sure, but it's why I asked what's the goal in my
very first message in this thread.
We need some sort of logical counter, such as a vector clock, which
might be an independent counter on each node but can perfectly track
the causal order. For example, if NodeA observes an operation from
NodeB with a counter value of X, NodeA will adjust its counter to X+1.
This ensures that if NodeA has seen an operation from NodeB, its next
operation will appear to have occurred after NodeB's operation.I admit that I haven't fully thought through how we could design such
version tracking in our logical replication protocol or how it would
fit into our system. However, my point is that we need to consider
something beyond commit timestamps to achieve reliable ordering.
I can't really respond to this as there's no suggestion how it would be
implemented in the patch discussed in this thread.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
On Mon, Jun 17, 2024 at 4:18 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
On 6/14/24 13:29, Amit Kapila wrote:
On Fri, Jun 14, 2024 at 12:10 AM Robert Haas <robertmhaas@gmail.com> wrote:
On Thu, May 23, 2024 at 2:37 AM shveta malik <shveta.malik@gmail.com> wrote:
c) update_deleted: The row with the same value as that incoming
update's key does not exist. The row is already deleted. This conflict
type is generated only if the deleted row is still detectable i.e., it
is not removed by VACUUM yet. If the row is removed by VACUUM already,
it cannot detect this conflict. It will detect it as update_missing
and will follow the default or configured resolver of update_missing
itself.I think this design is categorically unacceptable. It amounts to
designing a feature that works except when it doesn't. I'm not exactly
sure how the proposal should be changed to avoid depending on the
timing of VACUUM, but I think it's absolutely not OK to depend on the
timing of VACUUm -- or, really, this is going to depend on the timing
of HOT-pruning, which will often happen almost instantly.Agreed, above Tomas has speculated to have a way to avoid vacuum
cleaning dead tuples until the required changes are received and
applied. Shveta also mentioned another way to have deads-store (say a
table where deleted rows are stored for resolution) [1] which is
similar to a technique used by some other databases. There is an
agreement to not rely on Vacuum to detect such a conflict but the
alternative is not clear.I'm not sure I'd say I "speculated" about it - it's not like we don't
have ways to hold off cleanup for a while for various reasons
(long-running query, replication slot, hot-standby feedback, ...).How exactly would that be implemented I don't know, but it seems like a
far simpler approach than inventing a new "dead store". It'd need logic
to let the vacuum to cleanup the stuff no longer needed, but so would
the dead store I think.
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;
Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row? The one factor could be time,
say we define a new parameter vacuum_committs_age which would indicate
that we will allow rows to be removed only if the modified time of the
tuple as indicated by committs module is greater than the
vacuum_committs_age. This needs more analysis if we want to pursue
this direction.
OTOH, in the existing mechanisms, there is a common factor among all
which is that we know that there is some event that requires data to
be present. For example, with a long-running query, we know that the
deleted/updated row is still visible for some running query. For
replication slots, we know that the client will acknowledge the
feedback in terms of LSN using which we can allow vacuum to remove
rows. Similar to these hot_standby_feedback allows the vacuum to
prevent row removal based on current activity (the xid horizons
required by queries on standby) on hot_standby.
Currently, we are thinking to consider such
a conflict type as update_missing (The row with the same value as that
incoming update's key does not exist.). This is how the current HEAD
code behaves and LOGs the information (logical replication did not
find row to be updated ..).I thought the agreement was we need both conflict types to get sensible
behavior, so proceeding with just the update_missing (mostly because we
don't know how to detect these conflicts reliably) seems like maybe not
be the right direction ...
Fair enough. I am also not in favor of ignoring this but if as a first
step, we want to improve our current conflict detection mechanism and
provide the stats or conflict information in some catalog or view, we
can do that even if update_delete is not detected. For example, as of
now, we only detect update_missing and simply LOG it at DEBUG1 level.
Additionally, we can detect update_differ (the row updated by a
different origin) and have some stats. We seem to have some agreement
that conflict detection and stats about the same could be the first
step.
--
With Regards,
Amit Kapila.
On Mon, Jun 17, 2024 at 5:38 AM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:
The issue with using commit timestamps is that, when multiple nodes
are involved, the commit timestamp won't accurately represent the
actual order of operations. There's no reliable way to determine the
perfect order of each operation happening on different nodes roughly
simultaneously unless we use some globally synchronized counter.
Generally, that order might not cause real issues unless one operation
is triggered by a previous operation, and relying solely on physical
timestamps would not detect that correctly.This whole conflict detection / resolution proposal is based on using
commit timestamps. Aren't you suggesting it can't really work with
commit timestamps?FWIW there are ways to builds distributed consistency with timestamps,
as long as it's monotonic - e.g. clock-SI does that. It's not perfect,
but it shows it's possible.
Hmm, I see that clock-SI does this by delaying the transaction when it
detects the clock skew.
However, I'm not we have to go there - it depends on what the goal is.
For a one-directional replication (multiple nodes replicating to the
same target) it might be sufficient if the conflict resolution is
"deterministic" (e.g. not dependent on the order in which the changes
are applied). I'm not sure, but it's why I asked what's the goal in my
very first message in this thread.
I'm not completely certain about this. Even in one directional
replication if multiple nodes are sending data how can we guarantee
determinism in the presence of clock skew if we are not using some
other mechanism like logical counters or something like what clock-SI
is doing? I don't want to insist on using any specific solution here.
However, I noticed that we haven't addressed how we plan to manage
clock skew, which is my primary concern. I believe that if multiple
nodes are involved and we're receiving data from them with
unsynchronized clocks, ensuring determinism about their order will
require us to take some measures to handle that.
We need some sort of logical counter, such as a vector clock, which
might be an independent counter on each node but can perfectly track
the causal order. For example, if NodeA observes an operation from
NodeB with a counter value of X, NodeA will adjust its counter to X+1.
This ensures that if NodeA has seen an operation from NodeB, its next
operation will appear to have occurred after NodeB's operation.I admit that I haven't fully thought through how we could design such
version tracking in our logical replication protocol or how it would
fit into our system. However, my point is that we need to consider
something beyond commit timestamps to achieve reliable ordering.I can't really respond to this as there's no suggestion how it would be
implemented in the patch discussed in this thread.
No worries, I'll consider whether finding such a solution is feasible
for our situation. Thank you!
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model. However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution.
I think the unbounded size of the vector could be a problem to store
for each event. However, while researching previous discussions, it
came to our notice that we have discussed this topic in the past as
well in the context of standbys. For recovery_min_apply_delay, we
decided the clock skew is not a problem as the settings of this
parameter are much larger than typical time deviations between servers
as mentioned in docs. Similarly for casual reads [1]/messages/by-id/CAEepm=1iiEzCVLD=RoBgtZSyEY1CR-Et7fRc9prCZ9MuTz3pWg@mail.gmail.com, there was a
proposal to introduce max_clock_skew parameter and suggesting the user
to make sure to have NTP set up correctly. We have tried to check
other databases (like Ora and BDR) where CDR is implemented but didn't
find anything specific to clock skew. So, I propose to go with a GUC
like max_clock_skew such that if the difference of time between the
incoming transaction's commit time and the local time is more than
max_clock_skew then we raise an ERROR. It is not clear to me that
putting bigger effort into clock skew is worth especially when other
systems providing CDR feature (like Ora or BDR) for decades have not
done anything like vector clocks. It is possible that this is less of
a problem w.r.t CDR and just detecting the anomaly in clock skew is
good enough.
[1]: /messages/by-id/CAEepm=1iiEzCVLD=RoBgtZSyEY1CR-Et7fRc9prCZ9MuTz3pWg@mail.gmail.com
--
With Regards,
Amit Kapila.
On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row?
The problem arises because T2 and T3 might be applied out of order on
some nodes. Once either one of them has been applied on every node, no
further conflicts are possible.
--
Robert Haas
EDB: http://www.enterprisedb.com
On Thursday, June 13, 2024 2:11 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Hi,
On Wed, Jun 5, 2024 at 3:32 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
wrote:This time at PGconf.dev[1], we had some discussions regarding this
project. The proposed approach is to split the work into two main
components. The first part focuses on conflict detection, which aims
to identify and report conflicts in logical replication. This feature
will enable users to monitor the unexpected conflicts that may occur.
The second part involves the actual conflict resolution. Here, we will
provide built-in resolutions for each conflict and allow user to
choose which resolution will be used for which conflict(as described
in the initial email of this thread).I agree with this direction that we focus on conflict detection (and
logging) first and then develop conflict resolution on top of that.
Thanks for your reply !
Of course, we are open to alternative ideas and suggestions, and the
strategy above can be changed based on ongoing discussions and
feedback received.Here is the patch of the first part work, which adds a new parameter
detect_conflict for CREATE and ALTER subscription commands. This new
parameter will decide if subscription will go for conflict detection.
By default, conflict detection will be off for a subscription.When conflict detection is enabled, additional logging is triggered in
the following conflict scenarios:* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.While there exist other conflict types in logical replication, such as
an incoming insert conflicting with an existing row due to a primary
key or unique index, these cases already result in constraint violation errors.What does detect_conflict being true actually mean to users? I understand that
detect_conflict being true could introduce some overhead to detect conflicts.
But in terms of conflict detection, even if detect_confict is false, we detect
some conflicts such as concurrent inserts with the same key. Once we
introduce the complete conflict detection feature, I'm not sure there is a case
where a user wants to detect only some particular types of conflict.Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.I feel that we should log all types of conflict in an uniform way. For example,
with detect_conflict being true, the update_differ conflict is reported as
"conflict %s detected on relation "%s"", whereas concurrent inserts with the
same key is reported as "duplicate key value violates unique constraint "%s"",
which could confuse users.
Do you mean it's ok to add a pre-check before applying the INSERT, which will
verify if the remote tuple violates any unique constraints, and if it violates
then we log a conflict message ? I thought about this but was slightly
worried about the extra cost it would bring. OTOH, if we think it's acceptable,
we could do that since the cost is there only when detect_conflict is enabled.
I also thought of logging such a conflict message in pg_catch(), but I think we
lack some necessary info(relation, index name, column name) at the catch block.
Best Regards,
Hou zj
On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model. However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution.I think the unbounded size of the vector could be a problem to store
for each event. However, while researching previous discussions, it
came to our notice that we have discussed this topic in the past as
well in the context of standbys. For recovery_min_apply_delay, we
decided the clock skew is not a problem as the settings of this
parameter are much larger than typical time deviations between servers
as mentioned in docs. Similarly for casual reads [1], there was a
proposal to introduce max_clock_skew parameter and suggesting the user
to make sure to have NTP set up correctly. We have tried to check
other databases (like Ora and BDR) where CDR is implemented but didn't
find anything specific to clock skew. So, I propose to go with a GUC
like max_clock_skew such that if the difference of time between the
incoming transaction's commit time and the local time is more than
max_clock_skew then we raise an ERROR. It is not clear to me that
putting bigger effort into clock skew is worth especially when other
systems providing CDR feature (like Ora or BDR) for decades have not
done anything like vector clocks. It is possible that this is less of
a problem w.r.t CDR and just detecting the anomaly in clock skew is
good enough.
I believe that if we've accepted this solution elsewhere, then we can
also consider the same. Basically, we're allowing the application to
set its tolerance for clock skew. And, if the skew exceeds that
tolerance, it's the application's responsibility to synchronize;
otherwise, an error will occur. This approach seems reasonable.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jun 18, 2024 at 10:17 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model. However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution.I think the unbounded size of the vector could be a problem to store
for each event. However, while researching previous discussions, it
came to our notice that we have discussed this topic in the past as
well in the context of standbys. For recovery_min_apply_delay, we
decided the clock skew is not a problem as the settings of this
parameter are much larger than typical time deviations between servers
as mentioned in docs. Similarly for casual reads [1], there was a
proposal to introduce max_clock_skew parameter and suggesting the user
to make sure to have NTP set up correctly. We have tried to check
other databases (like Ora and BDR) where CDR is implemented but didn't
find anything specific to clock skew. So, I propose to go with a GUC
like max_clock_skew such that if the difference of time between the
incoming transaction's commit time and the local time is more than
max_clock_skew then we raise an ERROR. It is not clear to me that
putting bigger effort into clock skew is worth especially when other
systems providing CDR feature (like Ora or BDR) for decades have not
done anything like vector clocks. It is possible that this is less of
a problem w.r.t CDR and just detecting the anomaly in clock skew is
good enough.I believe that if we've accepted this solution elsewhere, then we can
also consider the same. Basically, we're allowing the application to
set its tolerance for clock skew. And, if the skew exceeds that
tolerance, it's the application's responsibility to synchronize;
otherwise, an error will occur. This approach seems reasonable.
This model can be further extended by making the apply worker wait if
the remote transaction's commit_ts is greater than the local
timestamp. This ensures that no local transactions occurring after the
remote transaction appear to have happened earlier due to clock skew
instead we make them happen before the remote transaction by delaying
the remote transaction apply. Essentially, by having the remote
application wait until the local timestamp matches the remote
transaction's timestamp, we ensure that the remote transaction, which
seems to occur after concurrent local transactions due to clock skew,
is actually applied after those transactions.
With this model, there should be no ordering errors from the
application's perspective as well if synchronous commit is enabled.
The transaction initiated by the publisher cannot be completed until
it is applied to the synchronous subscriber. This ensures that if the
subscriber's clock is lagging behind the publisher's clock, the
transaction will not be applied until the subscriber's local clock is
in sync, preventing the transaction from being completed out of order.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Mon, Jun 17, 2024 at 8:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row?The problem arises because T2 and T3 might be applied out of order on
some nodes. Once either one of them has been applied on every node, no
further conflicts are possible.
If we decide to skip the update whether the row is missing or deleted,
we indeed reach the same end result regardless of the order of T2, T3,
and Vacuum. Here's how it looks in each case:
Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row
-> end result we do not have a row.
Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result
we do not have a row.
Case 3: T1, T3, T2 -> deleted the row -> end result we do not have a row.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jun 18, 2024 at 11:54 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Jun 17, 2024 at 8:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row?The problem arises because T2 and T3 might be applied out of order on
some nodes. Once either one of them has been applied on every node, no
further conflicts are possible.If we decide to skip the update whether the row is missing or deleted,
we indeed reach the same end result regardless of the order of T2, T3,
and Vacuum. Here's how it looks in each case:Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row
-> end result we do not have a row.
Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result
we do not have a row.
Case 3: T1, T3, T2 -> deleted the row -> end result we do not have a row.
In case 3, how can deletion be successful? The row required to be
deleted has already been updated.
--
With Regards,
Amit Kapila.
On Tue, Jun 18, 2024 at 12:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:54 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Jun 17, 2024 at 8:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row?The problem arises because T2 and T3 might be applied out of order on
some nodes. Once either one of them has been applied on every node, no
further conflicts are possible.If we decide to skip the update whether the row is missing or deleted,
we indeed reach the same end result regardless of the order of T2, T3,
and Vacuum. Here's how it looks in each case:Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row
-> end result we do not have a row.
Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result
we do not have a row.
Case 3: T1, T3, T2 -> deleted the row -> end result we do not have a row.In case 3, how can deletion be successful? The row required to be
deleted has already been updated.
Hmm, I was considering this case in the example given by you above[1],
so we have updated some fields of the row with id=1, isn't this row
still detectable by the delete because delete will find this by id=1
as we haven't updated the id? I was making the point w.r.t. the
example used above.
[1]:
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jun 18, 2024 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 12:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:54 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Jun 17, 2024 at 8:51 PM Robert Haas <robertmhaas@gmail.com> wrote:
On Mon, Jun 17, 2024 at 1:42 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
The difference w.r.t the existing mechanisms for holding deleted data
is that we don't know whether we need to hold off the vacuum from
cleaning up the rows because we can't say with any certainty whether
other nodes will perform any conflicting operations in the future.
Using the example we discussed,
Node A:
T1: INSERT INTO t (id, value) VALUES (1,1);
T2: DELETE FROM t WHERE id = 1;Node B:
T3: UPDATE t SET value = 2 WHERE id = 1;Say the order of receiving the commands is T1-T2-T3. We can't predict
whether we will ever get T-3, so on what basis shall we try to prevent
vacuum from removing the deleted row?The problem arises because T2 and T3 might be applied out of order on
some nodes. Once either one of them has been applied on every node, no
further conflicts are possible.If we decide to skip the update whether the row is missing or deleted,
we indeed reach the same end result regardless of the order of T2, T3,
and Vacuum. Here's how it looks in each case:Case 1: T1, T2, Vacuum, T3 -> Skip the update for a non-existing row
-> end result we do not have a row.
Case 2: T1, T2, T3 -> Skip the update for a deleted row -> end result
we do not have a row.
Case 3: T1, T3, T2 -> deleted the row -> end result we do not have a row.In case 3, how can deletion be successful? The row required to be
deleted has already been updated.Hmm, I was considering this case in the example given by you above[1],
so we have updated some fields of the row with id=1, isn't this row
still detectable by the delete because delete will find this by id=1
as we haven't updated the id? I was making the point w.r.t. the
example used above.
Your point is correct w.r.t the example but I responded considering a
general update-delete ordering. BTW, it is not clear to me how
update_delete conflict will be handled with what Robert and you are
saying. I'll try to say what I understood. If we assume that there are
two nodes A & B as mentioned in the above example and DELETE has
applied on both nodes, now say UPDATE has been performed on node B
then irrespective of whether we consider the conflict as update_delete
or update_missing, the data will remain same on both nodes. So, in
such a case, we don't need to bother differentiating between those two
types of conflicts. Is that what we can interpret from above?
--
With Regards,
Amit Kapila.
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 10:17 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Mon, Jun 17, 2024 at 3:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 12, 2024 at 10:03 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 11, 2024 at 7:44 PM Tomas Vondra
<tomas.vondra@enterprisedb.com> wrote:Yes, that's correct. However, many cases could benefit from the
update_deleted conflict type if it can be implemented reliably. That's
why we wanted to give it a try. But if we can't achieve predictable
results with it, I'm fine to drop this approach and conflict_type. We
can consider a better design in the future that doesn't depend on
non-vacuumed entries and provides a more robust method for identifying
deleted rows.I agree having a separate update_deleted conflict would be beneficial,
I'm not arguing against that - my point is actually that I think this
conflict type is required, and that it needs to be detected reliably.When working with a distributed system, we must accept some form of
eventual consistency model. However, it's essential to design a
predictable and acceptable behavior. For example, if a change is a
result of a previous operation (such as an update on node B triggered
after observing an operation on node A), we can say that the operation
on node A happened before the operation on node B. Conversely, if
operations on nodes A and B are independent, we consider them
concurrent.In distributed systems, clock skew is a known issue. To establish a
consistency model, we need to ensure it guarantees the
"happens-before" relationship. Consider a scenario with three nodes:
NodeA, NodeB, and NodeC. If NodeA sends changes to NodeB, and
subsequently NodeB makes changes, and then both NodeA's and NodeB's
changes are sent to NodeC, the clock skew might make NodeB's changes
appear to have occurred before NodeA's changes. However, we should
maintain data that indicates NodeB's changes were triggered after
NodeA's changes arrived at NodeB. This implies that logically, NodeB's
changes happened after NodeA's changes, despite what the timestamps
suggest.A common method to handle such cases is using vector clocks for
conflict resolution.I think the unbounded size of the vector could be a problem to store
for each event. However, while researching previous discussions, it
came to our notice that we have discussed this topic in the past as
well in the context of standbys. For recovery_min_apply_delay, we
decided the clock skew is not a problem as the settings of this
parameter are much larger than typical time deviations between servers
as mentioned in docs. Similarly for casual reads [1], there was a
proposal to introduce max_clock_skew parameter and suggesting the user
to make sure to have NTP set up correctly. We have tried to check
other databases (like Ora and BDR) where CDR is implemented but didn't
find anything specific to clock skew. So, I propose to go with a GUC
like max_clock_skew such that if the difference of time between the
incoming transaction's commit time and the local time is more than
max_clock_skew then we raise an ERROR. It is not clear to me that
putting bigger effort into clock skew is worth especially when other
systems providing CDR feature (like Ora or BDR) for decades have not
done anything like vector clocks. It is possible that this is less of
a problem w.r.t CDR and just detecting the anomaly in clock skew is
good enough.I believe that if we've accepted this solution elsewhere, then we can
also consider the same. Basically, we're allowing the application to
set its tolerance for clock skew. And, if the skew exceeds that
tolerance, it's the application's responsibility to synchronize;
otherwise, an error will occur. This approach seems reasonable.This model can be further extended by making the apply worker wait if
the remote transaction's commit_ts is greater than the local
timestamp. This ensures that no local transactions occurring after the
remote transaction appear to have happened earlier due to clock skew
instead we make them happen before the remote transaction by delaying
the remote transaction apply. Essentially, by having the remote
application wait until the local timestamp matches the remote
transaction's timestamp, we ensure that the remote transaction, which
seems to occur after concurrent local transactions due to clock skew,
is actually applied after those transactions.With this model, there should be no ordering errors from the
application's perspective as well if synchronous commit is enabled.
The transaction initiated by the publisher cannot be completed until
it is applied to the synchronous subscriber. This ensures that if the
subscriber's clock is lagging behind the publisher's clock, the
transaction will not be applied until the subscriber's local clock is
in sync, preventing the transaction from being completed out of order.
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.
Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.
Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.
When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.
2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.
Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.
----------
In case 1, the local change which was otherwise triggered later than
the remote change is overwritten by remote change. And in Case2, it
results in data divergence. Is this behaviour in both cases expected?
Or am I getting the wait logic wrong? Thoughts?
thanks
Shveta
On Tue, Jun 18, 2024 at 7:44 AM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
On Thursday, June 13, 2024 2:11 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Hi,
On Wed, Jun 5, 2024 at 3:32 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com>
wrote:This time at PGconf.dev[1], we had some discussions regarding this
project. The proposed approach is to split the work into two main
components. The first part focuses on conflict detection, which aims
to identify and report conflicts in logical replication. This feature
will enable users to monitor the unexpected conflicts that may occur.
The second part involves the actual conflict resolution. Here, we will
provide built-in resolutions for each conflict and allow user to
choose which resolution will be used for which conflict(as described
in the initial email of this thread).I agree with this direction that we focus on conflict detection (and
logging) first and then develop conflict resolution on top of that.Thanks for your reply !
Of course, we are open to alternative ideas and suggestions, and the
strategy above can be changed based on ongoing discussions and
feedback received.Here is the patch of the first part work, which adds a new parameter
detect_conflict for CREATE and ALTER subscription commands. This new
parameter will decide if subscription will go for conflict detection.
By default, conflict detection will be off for a subscription.When conflict detection is enabled, additional logging is triggered in
the following conflict scenarios:* updating a row that was previously modified by another origin.
* The tuple to be updated is not found.
* The tuple to be deleted is not found.While there exist other conflict types in logical replication, such as
an incoming insert conflicting with an existing row due to a primary
key or unique index, these cases already result in constraint violation errors.What does detect_conflict being true actually mean to users? I understand that
detect_conflict being true could introduce some overhead to detect conflicts.
But in terms of conflict detection, even if detect_confict is false, we detect
some conflicts such as concurrent inserts with the same key. Once we
introduce the complete conflict detection feature, I'm not sure there is a case
where a user wants to detect only some particular types of conflict.Therefore, additional conflict detection for these cases is currently
omitted to minimize potential overhead. However, the pre-detection for
conflict in these error cases is still essential to support automatic
conflict resolution in the future.I feel that we should log all types of conflict in an uniform way. For example,
with detect_conflict being true, the update_differ conflict is reported as
"conflict %s detected on relation "%s"", whereas concurrent inserts with the
same key is reported as "duplicate key value violates unique constraint "%s"",
which could confuse users.Do you mean it's ok to add a pre-check before applying the INSERT, which will
verify if the remote tuple violates any unique constraints, and if it violates
then we log a conflict message ? I thought about this but was slightly
worried about the extra cost it would bring. OTOH, if we think it's acceptable,
we could do that since the cost is there only when detect_conflict is enabled.I also thought of logging such a conflict message in pg_catch(), but I think we
lack some necessary info(relation, index name, column name) at the catch block.
Can't we use/extend existing 'apply_error_callback_arg' for this purpose?
--
With Regards,
Amit Kapila.
On Tue, Jun 11, 2024 at 3:12 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
On Sat, Jun 8, 2024 at 3:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Jun 7, 2024 at 5:39 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:On Thu, Jun 6, 2024 at 5:16 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Here are more use cases of the "earliest_timestamp_wins" resolution method:
1) Applications where the record of first occurrence of an event is
important. For example, sensor based applications like earthquake
detection systems, capturing the first seismic wave's time is crucial.
2) Scheduling systems, like appointment booking, prioritize the
earliest request when handling concurrent ones.
3) In contexts where maintaining chronological order is important -
a) Social media platforms display comments ensuring that the
earliest ones are visible first.
b) Finance transaction processing systems rely on timestamps to
prioritize the processing of transactions, ensuring that the earliest
transaction is handled firstThanks for sharing examples. However, these scenarios would be handled by the application and not during replication. What we are discussing here is the timestamp when a row was updated/inserted/deleted (or rather when the transaction that updated row committed/became visible) and not a DML on column which is of type timestamp. Some implementations use a hidden timestamp column but that's different from a user column which captures timestamp of (say) an event. The conflict resolution will be based on the timestamp when that column's value was recorded in the database which may be different from the value of the column itself.
It depends on how these operations are performed. For example, the
appointment booking system could be prioritized via a transaction
updating a row with columns emp_name, emp_id, reserved, time_slot.
Now, if two employees at different geographical locations try to book
the calendar, the earlier transaction will win.I doubt that it would be that simple. The application will have to intervene and tell one of the employees that their reservation has failed. It looks natural that the first one to reserve the room should get the reservation, but implementing that is more complex than resolving a conflict in the database. In fact, mostly it will be handled outside database.
Sure, the application needs some handling but I have tried to explain
with a simple way that comes to my mind and how it can be realized
with db involved. This is a known conflict detection method but note
that I am not insisting to have "earliest_timestamp_wins". Even, if we
want this we can have a separate discussion on this and add it later.
If we use the transaction commit timestamp as basis for resolution, a transaction where multiple rows conflict may end up with different rows affected by that transaction being resolved differently. Say three transactions T1, T2 and T3 on separate origins with timestamps t1, t2, and t3 respectively changed rows r1, r2 and r2, r3 and r1, r4 respectively. Changes to r1 and r2 will conflict. Let's say T2 and T3 are applied first and then T1 is applied. If t2 < t1 < t3, r1 will end up with version of T3 and r2 will end up with version of T1 after applying all the three transactions.
Are you telling the results based on latest_timestamp_wins? If so,
then it is correct. OTOH, if the user has configured
"earliest_timestamp_wins" resolution method, then we should end up
with a version of r1 from T1 because t1 < t3. Also, due to the same
reason, we should have version r2 from T2.Would that introduce an inconsistency between r1 and r2?
As per my understanding, this shouldn't be an inconsistency. Won't it
be true even when the transactions are performed on a single node with
the same timing?The inconsistency will arise irrespective of conflict resolution method. On a single system effects of whichever transaction runs last will be visible entirely. But in the example above the node where T1, T2, and T3 (from *different*) origins) are applied, we might end up with a situation where some changes from T1 are applied whereas some changes from T3 are applied.
I still think it will lead to the same result if all three T1, T2, T3
happen on the same node in the same order as you mentioned. Say, we
have a pre-existing table with rows r1, r2, r3, r4. Now, if we use the
order of transactions to be applied on the same node based on t2 < t1
< t3. First T2 will be applied, so for now, r1 is a pre-existing
version and r2 is from T2. Next, when T1 is performed, both r1 and r2
are from T1. Lastly, when T3 is applied, r1 will be from T3 and r2
will be from T1. This is what you mentioned will happen after conflict
resolution in the above example.
--
With Regards,
Amit Kapila.
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.
For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.
I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
node which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2 and the timestamp will also show the same at any other
node if they receive these 2 changes.
The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.
2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.
Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2
IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.
So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame. Generally, the ideal configuration for
max_clock_skew should be in multiple of the network round trip time.
Assuming this configuration, we wouldn’t encounter this problem
because for change-2 to be caused by change-1, the client would need
to get confirmation of change-1 and then trigger change-2, which would
take at least 2-3 network round trips.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.
Do you mean "the change that happened at 10:18 as change-2"
I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
node
Do you mean "we delayed applying change-1 on the local node."
which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2
Do you mean "So it's fine that we perform change-2 before change-1"
and the timestamp will also show the same at any other
node if they receive these 2 changes.The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.
Considering the above corrections as base, I agree with this.
2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame.
Agree. I had the same thoughts, and wanted to confirm my understanding.
Generally, the ideal configuration for
max_clock_skew should be in multiple of the network round trip time.
Assuming this configuration, we wouldn’t encounter this problem
because for change-2 to be caused by change-1, the client would need
to get confirmation of change-1 and then trigger change-2, which would
take at least 2-3 network round trips.
thanks
Shveta
On Wed, Jun 19, 2024 at 12:03 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:
I doubt that it would be that simple. The application will have to
intervene and tell one of the employees that their reservation has failed.
It looks natural that the first one to reserve the room should get the
reservation, but implementing that is more complex than resolving a
conflict in the database. In fact, mostly it will be handled outside
database.Sure, the application needs some handling but I have tried to explain
with a simple way that comes to my mind and how it can be realized
with db involved. This is a known conflict detection method but note
that I am not insisting to have "earliest_timestamp_wins". Even, if we
want this we can have a separate discussion on this and add it later.
It will be good to add a minimal set of conflict resolution strategies to
begin with, while designing the feature for extensibility. I imagine the
first version might just detect the conflict and throw error or do nothing.
That's already two simple conflict resolution strategies with minimal
efforts. We can add more complicated ones incrementally.
The inconsistency will arise irrespective of conflict resolution method.
On a single system effects of whichever transaction runs last will be
visible entirely. But in the example above the node where T1, T2, and T3
(from *different*) origins) are applied, we might end up with a situation
where some changes from T1 are applied whereas some changes from T3 are
applied.I still think it will lead to the same result if all three T1, T2, T3
happen on the same node in the same order as you mentioned. Say, we
have a pre-existing table with rows r1, r2, r3, r4. Now, if we use the
order of transactions to be applied on the same node based on t2 < t1
< t3. First T2 will be applied, so for now, r1 is a pre-existing
version and r2 is from T2. Next, when T1 is performed, both r1 and r2
are from T1. Lastly, when T3 is applied, r1 will be from T3 and r2
will be from T1. This is what you mentioned will happen after conflict
resolution in the above example.
You are right. It won't affect the consistency. The contents of transaction
on each node might vary after application depending upon the changes that
conflict resolver makes; but the end result will be the same.
--
Best Wishes,
Ashutosh Bapat
On Wed, Jun 19, 2024 at 2:36 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.
Oops lot of mistakes in the usage of change-1 and change-2, sorry about that.
For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.Do you mean "the change that happened at 10:18 as change-2"
Right
I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
nodeDo you mean "we delayed applying change-1 on the local node."
Right
which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2Do you mean "So it's fine that we perform change-2 before change-1"
Right
and the timestamp will also show the same at any other
node if they receive these 2 changes.The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.Considering the above corrections as base, I agree with this.
+1
2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame.Agree. I had the same thoughts, and wanted to confirm my understanding.
Okay
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jun 19, 2024 at 2:51 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
On Wed, Jun 19, 2024 at 12:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
I doubt that it would be that simple. The application will have to intervene and tell one of the employees that their reservation has failed. It looks natural that the first one to reserve the room should get the reservation, but implementing that is more complex than resolving a conflict in the database. In fact, mostly it will be handled outside database.
Sure, the application needs some handling but I have tried to explain
with a simple way that comes to my mind and how it can be realized
with db involved. This is a known conflict detection method but note
that I am not insisting to have "earliest_timestamp_wins". Even, if we
want this we can have a separate discussion on this and add it later.It will be good to add a minimal set of conflict resolution strategies to begin with, while designing the feature for extensibility. I imagine the first version might just detect the conflict and throw error or do nothing. That's already two simple conflict resolution strategies with minimal efforts. We can add more complicated ones incrementally.
Agreed, splitting the work into multiple patches would help us to
finish the easier ones first.
I have thought to divide it such that in the first patch, we detect
conflicts like 'insert_exists', 'update_differ', 'update_missing', and
'delete_missing' (the definition of each could be found in the initial
email [1]/messages/by-id/CAJpy0uD0-DpYVMtsxK5R=zszXauZBayQMAYET9sWr_w0CNWXxQ@mail.gmail.com) and throw an ERROR or write them in LOG. Various people
agreed to have this as a separate committable work [2]/messages/by-id/CAD21AoAa6JzqhXY02uNUPb-aTozu2RY9nMdD1=TUh+FpskkYtw@mail.gmail.com. This can help
users to detect and monitor the conflicts in a better way. I have
intentionally skipped update_deleted as it would require more
infrastructure and it would be helpful even without that.
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3]/messages/by-id/CAJpy0uD0-DpYVMtsxK5R=zszXauZBayQMAYET9sWr_w0CNWXxQ@mail.gmail.com[4]https://github.com/2ndquadrant/pglogical?tab=readme-ov-file#conflicts for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.
In the third patch, we can add monitoring capability for conflicts and
resolutions as mentioned by Jonathan [5]/messages/by-id/1eb9242f-dcb6-45c3-871c-98ec324e03ef@postgresql.org. Here, we can have stats like
how many conflicts of a particular type have happened.
In the meantime, we can keep discussing and try to reach a consensus
on the timing-related resolution strategy like 'last_update_wins' and
the conflict strategy 'update_deleted'.
If we agree on the above, some of the work, especially the first one,
could even be discussed in a separate thread.
Thoughts?
[1]: /messages/by-id/CAJpy0uD0-DpYVMtsxK5R=zszXauZBayQMAYET9sWr_w0CNWXxQ@mail.gmail.com
[2]: /messages/by-id/CAD21AoAa6JzqhXY02uNUPb-aTozu2RY9nMdD1=TUh+FpskkYtw@mail.gmail.com
[3]: /messages/by-id/CAJpy0uD0-DpYVMtsxK5R=zszXauZBayQMAYET9sWr_w0CNWXxQ@mail.gmail.com
[4]: https://github.com/2ndquadrant/pglogical?tab=readme-ov-file#conflicts
[5]: /messages/by-id/1eb9242f-dcb6-45c3-871c-98ec324e03ef@postgresql.org
--
With Regards,
Amit Kapila.
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 10:17 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I think the unbounded size of the vector could be a problem to store
for each event. However, while researching previous discussions, it
came to our notice that we have discussed this topic in the past as
well in the context of standbys. For recovery_min_apply_delay, we
decided the clock skew is not a problem as the settings of this
parameter are much larger than typical time deviations between servers
as mentioned in docs. Similarly for casual reads [1], there was a
proposal to introduce max_clock_skew parameter and suggesting the user
to make sure to have NTP set up correctly. We have tried to check
other databases (like Ora and BDR) where CDR is implemented but didn't
find anything specific to clock skew. So, I propose to go with a GUC
like max_clock_skew such that if the difference of time between the
incoming transaction's commit time and the local time is more than
max_clock_skew then we raise an ERROR. It is not clear to me that
putting bigger effort into clock skew is worth especially when other
systems providing CDR feature (like Ora or BDR) for decades have not
done anything like vector clocks. It is possible that this is less of
a problem w.r.t CDR and just detecting the anomaly in clock skew is
good enough.I believe that if we've accepted this solution elsewhere, then we can
also consider the same. Basically, we're allowing the application to
set its tolerance for clock skew. And, if the skew exceeds that
tolerance, it's the application's responsibility to synchronize;
otherwise, an error will occur. This approach seems reasonable.This model can be further extended by making the apply worker wait if
the remote transaction's commit_ts is greater than the local
timestamp. This ensures that no local transactions occurring after the
remote transaction appear to have happened earlier due to clock skew
instead we make them happen before the remote transaction by delaying
the remote transaction apply. Essentially, by having the remote
application wait until the local timestamp matches the remote
transaction's timestamp, we ensure that the remote transaction, which
seems to occur after concurrent local transactions due to clock skew,
is actually applied after those transactions.With this model, there should be no ordering errors from the
application's perspective as well if synchronous commit is enabled.
The transaction initiated by the publisher cannot be completed until
it is applied to the synchronous subscriber. This ensures that if the
subscriber's clock is lagging behind the publisher's clock, the
transaction will not be applied until the subscriber's local clock is
in sync, preventing the transaction from being completed out of order.
As per the discussion, this idea will help us to resolve transaction
ordering issues due to clock skew. I was thinking of having two
variables max_clock_skew (indicates how much clock skew is
acceptable), max_clock_skew_options: ERROR, LOG, WAIT (indicates the
action we need to take once the clock skew is detected). There could
be multiple ways to provide these parameters, one is providing them as
GUCs, and another at the subscription or the table level. I am
thinking whether users would only like to care about a table or set of
tables or they would like to set such variables at the system level.
We already have an SKIP option (that allows us to skip the
transactions till a particular LSN) at the subscription level, so I am
wondering if there is a sense to provide these new parameters related
to conflict resolution also at the same level?
--
With Regards,
Amit Kapila.
On Thu, Jun 20, 2024 at 3:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 19, 2024 at 2:51 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:On Wed, Jun 19, 2024 at 12:03 PM Amit Kapila <amit.kapila16@gmail.com>
wrote:
I doubt that it would be that simple. The application will have to
intervene and tell one of the employees that their reservation has failed.
It looks natural that the first one to reserve the room should get the
reservation, but implementing that is more complex than resolving a
conflict in the database. In fact, mostly it will be handled outside
database.Sure, the application needs some handling but I have tried to explain
with a simple way that comes to my mind and how it can be realized
with db involved. This is a known conflict detection method but note
that I am not insisting to have "earliest_timestamp_wins". Even, if we
want this we can have a separate discussion on this and add it later.It will be good to add a minimal set of conflict resolution strategies
to begin with, while designing the feature for extensibility. I imagine the
first version might just detect the conflict and throw error or do nothing.
That's already two simple conflict resolution strategies with minimal
efforts. We can add more complicated ones incrementally.Agreed, splitting the work into multiple patches would help us to
finish the easier ones first.I have thought to divide it such that in the first patch, we detect
conflicts like 'insert_exists', 'update_differ', 'update_missing', and
'delete_missing' (the definition of each could be found in the initial
email [1]) and throw an ERROR or write them in LOG. Various people
agreed to have this as a separate committable work [2]. This can help
users to detect and monitor the conflicts in a better way. I have
intentionally skipped update_deleted as it would require more
infrastructure and it would be helpful even without that.
Since we are in the initial months of release, it will be good to take a
stock of whether the receiver receives all the information needed for most
(if not all) of the conflict detection and resolution strategies. If there
are any missing pieces, we may want to add those in PG18 so that improved
conflict detection and resolution on a higher version receiver can still
work.
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.In the third patch, we can add monitoring capability for conflicts and
resolutions as mentioned by Jonathan [5]. Here, we can have stats like
how many conflicts of a particular type have happened.
That looks like a plan. Thanks for chalking it out.
--
Best Wishes,
Ashutosh Bapat
On Thu, Jun 20, 2024 at 5:06 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:
On Thu, Jun 20, 2024 at 3:21 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jun 19, 2024 at 2:51 PM Ashutosh Bapat
<ashutosh.bapat.oss@gmail.com> wrote:On Wed, Jun 19, 2024 at 12:03 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
I doubt that it would be that simple. The application will have to intervene and tell one of the employees that their reservation has failed. It looks natural that the first one to reserve the room should get the reservation, but implementing that is more complex than resolving a conflict in the database. In fact, mostly it will be handled outside database.
Sure, the application needs some handling but I have tried to explain
with a simple way that comes to my mind and how it can be realized
with db involved. This is a known conflict detection method but note
that I am not insisting to have "earliest_timestamp_wins". Even, if we
want this we can have a separate discussion on this and add it later.It will be good to add a minimal set of conflict resolution strategies to begin with, while designing the feature for extensibility. I imagine the first version might just detect the conflict and throw error or do nothing. That's already two simple conflict resolution strategies with minimal efforts. We can add more complicated ones incrementally.
Agreed, splitting the work into multiple patches would help us to
finish the easier ones first.I have thought to divide it such that in the first patch, we detect
conflicts like 'insert_exists', 'update_differ', 'update_missing', and
'delete_missing' (the definition of each could be found in the initial
email [1]) and throw an ERROR or write them in LOG. Various people
agreed to have this as a separate committable work [2]. This can help
users to detect and monitor the conflicts in a better way. I have
intentionally skipped update_deleted as it would require more
infrastructure and it would be helpful even without that.Since we are in the initial months of release, it will be good to take a stock of whether the receiver receives all the information needed for most (if not all) of the conflict detection and resolution strategies. If there are any missing pieces, we may want to add those in PG18 so that improved conflict detection and resolution on a higher version receiver can still work.
Good point. This can help us to detect conflicts if required even when
we move to a higher version. As we continue to discuss/develop the
features, I hope we will be able to see any missing pieces.
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.In the third patch, we can add monitoring capability for conflicts and
resolutions as mentioned by Jonathan [5]. Here, we can have stats like
how many conflicts of a particular type have happened.That looks like a plan. Thanks for chalking it out.
Thanks!
--
With Regards,
Amit Kapila.
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.
Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1]/messages/by-id/OS0PR01MB57161006B8F2779F2C97318194D42@OS0PR01MB5716.jpnprd01.prod.outlook.com.
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1]/messages/by-id/OS0PR01MB57161006B8F2779F2C97318194D42@OS0PR01MB5716.jpnprd01.prod.outlook.com if you have any comments on patch001.
New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'
TODO: Once we get initial consensus on DDL commands, I will add
support for them in pg_dump/restore and will add doc.
------------
As suggested in [2]/messages/by-id/4738d098-6378-494e-9f88-9e3a85a5de82@enterprisedb.com and above, it seems logical to have table-specific
resolvers configuration along with global one.
Here is the proposal for table level resolvers:
1) We can provide support for table level resolvers using ALTER TABLE:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER
<resolver2> on <conflict_type2>, ...;
Reset can be done using:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on
<conflict_type2>, ...;
Above commands will save/remove configuration in/from the new system
catalog pg_conflict_rel.
2) Table level configuration (if any) will be given preference over
global ones. The tables not having table-specific resolvers will use
global configured ones.
3) If the table is a partition table, then resolvers created for the
parent will be inherited by all child partition tables. Multiple
resolver entries will be created, one for each child partition in the
system catalog (similar to constraints).
4) Users can also configure explicit resolvers for child partitions.
In such a case, child's resolvers will override inherited resolvers
(if any).
5) Any attempt to RESET (remove) inherited resolvers on the child
partition table *alone* will result in error: "cannot reset inherited
resolvers" (similar to constraints). But RESET of explicit created
resolvers (non-inherited ones) will be permitted for child partitions.
On RESET, the resolver configuration will not fallback to the
inherited resolver again. Users need to explicitly configure new
resolvers for the child partition tables (after RESET) if needed.
6) Removal/Reset of resolvers on parent will remove corresponding
"inherited" resolvers on all the child partitions as well. If any
child has overridden inherited resolvers earlier, those will stay.
7) For 'ALTER TABLE parent ATTACH PARTITION child'; if 'child' has its
own resolvers set, those will not be overridden. But if it does not
have resolvers set, it will inherit from the parent table. This will
mean, for say out of 5 conflict_types, if the child table has
resolvers configured for any 2, 'attach' will retain those; for the
rest 3, it will inherit from the parent (if any).
8) Detach partition will not remove inherited resolvers, it will just
mark them 'non inherited' (similar to constraints).
Thoughts?
------------
[1]: /messages/by-id/OS0PR01MB57161006B8F2779F2C97318194D42@OS0PR01MB5716.jpnprd01.prod.outlook.com
[2]: /messages/by-id/4738d098-6378-494e-9f88-9e3a85a5de82@enterprisedb.com
thanks
Shveta
Attachments:
v1-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v1-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 206583ed5ee755009e674222243bf3af387d7fc2 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v1 1/2] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that iolates a NOT DEFERRABLE unique constraint.
update_differ: updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ conflict can only be detected when track_commit_timestamp is
enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 55 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 187 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 50 ++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 44 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 ++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
25 files changed, 695 insertions(+), 151 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a63cc71efa..a9b6f293ea 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8034,6 +8034,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ce37fa6490 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,61 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index efb29adeb3..30393e6d67 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1360,7 +1360,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..2a2d188b19 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ingored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..f24f048054
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,187 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..af73e09b01 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2821,13 +2836,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3005,13 +3017,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..c6b67c692d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5140,6 +5150,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..02aa4a6f32 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..0472fe2e87 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6597,6 +6597,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..0bc9db991e
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..c64f17211d 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61ad417cde..f42efe12b7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From 79b28d6d918263312c8aeff5729d6adf33d739b9 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Mon, 24 Jun 2024 09:55:15 +0530
Subject: [PATCH v1 2/2] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 215 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/pg_conflict.dat | 20 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 +++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 50 ++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 27 +++
src/tools/pgindent/typedefs.list | 4 +
14 files changed, 467 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4d582950b7..987f3aee45 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index f24f048054..87b1ea0ab3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -27,6 +36,51 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -34,6 +88,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -185,3 +240,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..a3e5e10d97
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,20 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0bc9db991e..7eff93eff3 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -33,6 +34,41 @@ typedef enum
CT_DELETE_MISSING,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -40,5 +76,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..15c8c0cdba
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,50 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(4 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 969ced994f..4a4a1d9635 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..e3540d7ac8
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,27 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f42efe12b7..0fedd2eae3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,7 +465,11 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
+ConflictResolverNames
ConflictType
+ConflictTypeNames
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
On Mon, Jun 24, 2024 at 1:47 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1].
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1] if you have any comments on patch001.New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'
Does setting up resolvers have any meaning without subscriptions? I am
wondering whether we should allow to set up the resolvers at the
subscription level. One benefit is that users don't need to use a
different DDL to set up resolvers. The first patch gives a conflict
detection option at the subscription level, so it would be symmetrical
to provide a resolver at the subscription level. Yet another benefit
could be that it provides users facility to configure different
resolvers for a set of tables belonging to a particular
publication/node.
------------
As suggested in [2] and above, it seems logical to have table-specific
resolvers configuration along with global one.Here is the proposal for table level resolvers:
1) We can provide support for table level resolvers using ALTER TABLE:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER
<resolver2> on <conflict_type2>, ...;Reset can be done using:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on
<conflict_type2>, ...;Above commands will save/remove configuration in/from the new system
catalog pg_conflict_rel.2) Table level configuration (if any) will be given preference over
global ones. The tables not having table-specific resolvers will use
global configured ones.3) If the table is a partition table, then resolvers created for the
parent will be inherited by all child partition tables. Multiple
resolver entries will be created, one for each child partition in the
system catalog (similar to constraints).4) Users can also configure explicit resolvers for child partitions.
In such a case, child's resolvers will override inherited resolvers
(if any).5) Any attempt to RESET (remove) inherited resolvers on the child
partition table *alone* will result in error: "cannot reset inherited
resolvers" (similar to constraints). But RESET of explicit created
resolvers (non-inherited ones) will be permitted for child partitions.
On RESET, the resolver configuration will not fallback to the
inherited resolver again. Users need to explicitly configure new
resolvers for the child partition tables (after RESET) if needed.
Why so? If we can allow the RESET command to fallback to the inherited
resolver it would make the behavior consistent for the child table
where we don't have performed SET.
6) Removal/Reset of resolvers on parent will remove corresponding
"inherited" resolvers on all the child partitions as well. If any
child has overridden inherited resolvers earlier, those will stay.7) For 'ALTER TABLE parent ATTACH PARTITION child'; if 'child' has its
own resolvers set, those will not be overridden. But if it does not
have resolvers set, it will inherit from the parent table. This will
mean, for say out of 5 conflict_types, if the child table has
resolvers configured for any 2, 'attach' will retain those; for the
rest 3, it will inherit from the parent (if any).8) Detach partition will not remove inherited resolvers, it will just
mark them 'non inherited' (similar to constraints).
BTW, to keep the initial patch simple, can we prohibit setting
resolvers at the child table level? If we follow this, then we can
give an ERROR if the user tries to attach the table (with configured
resolvers) to an existing partitioned table.
--
With Regards,
Amit Kapila.
On Tue, Jun 25, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jun 24, 2024 at 1:47 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1].
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1] if you have any comments on patch001.New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'Does setting up resolvers have any meaning without subscriptions? I am
wondering whether we should allow to set up the resolvers at the
subscription level. One benefit is that users don't need to use a
different DDL to set up resolvers. The first patch gives a conflict
detection option at the subscription level, so it would be symmetrical
to provide a resolver at the subscription level. Yet another benefit
could be that it provides users facility to configure different
resolvers for a set of tables belonging to a particular
publication/node.
There can be multiple tables included in a publication with varying
business use-cases and thus may need different resolvers set, even
though they all are part of the same publication.
------------
As suggested in [2] and above, it seems logical to have table-specific
resolvers configuration along with global one.Here is the proposal for table level resolvers:
1) We can provide support for table level resolvers using ALTER TABLE:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER
<resolver2> on <conflict_type2>, ...;Reset can be done using:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on
<conflict_type2>, ...;Above commands will save/remove configuration in/from the new system
catalog pg_conflict_rel.2) Table level configuration (if any) will be given preference over
global ones. The tables not having table-specific resolvers will use
global configured ones.3) If the table is a partition table, then resolvers created for the
parent will be inherited by all child partition tables. Multiple
resolver entries will be created, one for each child partition in the
system catalog (similar to constraints).4) Users can also configure explicit resolvers for child partitions.
In such a case, child's resolvers will override inherited resolvers
(if any).5) Any attempt to RESET (remove) inherited resolvers on the child
partition table *alone* will result in error: "cannot reset inherited
resolvers" (similar to constraints). But RESET of explicit created
resolvers (non-inherited ones) will be permitted for child partitions.
On RESET, the resolver configuration will not fallback to the
inherited resolver again. Users need to explicitly configure new
resolvers for the child partition tables (after RESET) if needed.Why so? If we can allow the RESET command to fallback to the inherited
resolver it would make the behavior consistent for the child table
where we don't have performed SET.
Thought behind not making it fallback is since the user has done
'RESET', he may want to remove the resolver completely. We don't know
if he really wants to go back to the previous one. If he does, it is
easy to set it again. But if he does not, and we set the inherited
resolver again during 'RESET', there is no way he can drop that
inherited resolver alone on the child partition.
6) Removal/Reset of resolvers on parent will remove corresponding
"inherited" resolvers on all the child partitions as well. If any
child has overridden inherited resolvers earlier, those will stay.7) For 'ALTER TABLE parent ATTACH PARTITION child'; if 'child' has its
own resolvers set, those will not be overridden. But if it does not
have resolvers set, it will inherit from the parent table. This will
mean, for say out of 5 conflict_types, if the child table has
resolvers configured for any 2, 'attach' will retain those; for the
rest 3, it will inherit from the parent (if any).8) Detach partition will not remove inherited resolvers, it will just
mark them 'non inherited' (similar to constraints).BTW, to keep the initial patch simple, can we prohibit setting
resolvers at the child table level? If we follow this, then we can
give an ERROR if the user tries to attach the table (with configured
resolvers) to an existing partitioned table.
Okay, I will think about this if the patch becomes too complex.
thanks
Shveta
On Tue, Jun 25, 2024 at 3:39 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 25, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jun 24, 2024 at 1:47 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1].
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1] if you have any comments on patch001.New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'Does setting up resolvers have any meaning without subscriptions? I am
wondering whether we should allow to set up the resolvers at the
subscription level. One benefit is that users don't need to use a
different DDL to set up resolvers. The first patch gives a conflict
detection option at the subscription level, so it would be symmetrical
to provide a resolver at the subscription level. Yet another benefit
could be that it provides users facility to configure different
resolvers for a set of tables belonging to a particular
publication/node.There can be multiple tables included in a publication with varying
business use-cases and thus may need different resolvers set, even
though they all are part of the same publication.
Agreed but this is the reason we are planning to keep resolvers at the
table level. Here, I am asking to set resolvers at the subscription
level rather than at the global level.
------------
As suggested in [2] and above, it seems logical to have table-specific
resolvers configuration along with global one.Here is the proposal for table level resolvers:
1) We can provide support for table level resolvers using ALTER TABLE:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER
<resolver2> on <conflict_type2>, ...;Reset can be done using:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on
<conflict_type2>, ...;Above commands will save/remove configuration in/from the new system
catalog pg_conflict_rel.2) Table level configuration (if any) will be given preference over
global ones. The tables not having table-specific resolvers will use
global configured ones.3) If the table is a partition table, then resolvers created for the
parent will be inherited by all child partition tables. Multiple
resolver entries will be created, one for each child partition in the
system catalog (similar to constraints).4) Users can also configure explicit resolvers for child partitions.
In such a case, child's resolvers will override inherited resolvers
(if any).5) Any attempt to RESET (remove) inherited resolvers on the child
partition table *alone* will result in error: "cannot reset inherited
resolvers" (similar to constraints). But RESET of explicit created
resolvers (non-inherited ones) will be permitted for child partitions.
On RESET, the resolver configuration will not fallback to the
inherited resolver again. Users need to explicitly configure new
resolvers for the child partition tables (after RESET) if needed.Why so? If we can allow the RESET command to fallback to the inherited
resolver it would make the behavior consistent for the child table
where we don't have performed SET.Thought behind not making it fallback is since the user has done
'RESET', he may want to remove the resolver completely. We don't know
if he really wants to go back to the previous one. If he does, it is
easy to set it again. But if he does not, and we set the inherited
resolver again during 'RESET', there is no way he can drop that
inherited resolver alone on the child partition.
I see your point but normally RESET allows us to go back to the
default which in this case would be the resolver inherited from the
parent table.
--
With Regards,
Amit Kapila.
Please find the attached 'patch0003', which implements conflict
resolutions according to the global resolver settings.
Summary of Conflict Resolutions Implemented in 'patch0003':
INSERT Conflicts:
------------------------
1) Conflict Type: 'insert_exists'
Supported Resolutions:
a) 'remote_apply': Convert the INSERT to an UPDATE and apply.
b) 'keep_local': Ignore the incoming (conflicting) INSERT and retain
the local tuple.
c) 'error': The apply worker will error out and restart.
UPDATE Conflicts:
------------------------
1) Conflict Type: 'update_differ'
Supported Resolutions:
a) 'remote_apply': Apply the remote update.
b) 'keep_local': Skip the remote update and retain the local tuple.
c) 'error': The apply worker will error out and restart.
2) Conflict Type: 'update_missing'
Supported Resolutions:
a) 'apply_or_skip': Try to convert the UPDATE to an INSERT; if
unsuccessful, skip the remote update and continue.
b) 'apply_or_error': Try to convert the UPDATE to an INSERT; if
unsuccessful, error out.
c) 'skip': Skip the remote update and continue.
d) 'error': The apply worker will error out and restart.
DELETE Conflicts:
------------------------
1) Conflict Type: 'delete_missing'
Supported Resolutions:
a) 'skip': Skip the remote delete and continue.
b) 'error': The apply worker will error out and restart.
NOTE: With these basic resolution techniques, the patch does not aim
to ensure consistency across nodes, so data divergence is expected.
--
Thanks,
Nisha
Attachments:
v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From 83d0d11fbb7ab550f1d0c71647ac1631bb5b2e31 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Mon, 24 Jun 2024 09:55:15 +0530
Subject: [PATCH v1 2/3] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
insert_exists
update_differ
update_missing
delete_missing
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 215 ++++++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/pg_conflict.dat | 20 ++
src/include/catalog/pg_conflict.h | 43 +++++
src/include/nodes/parsenodes.h | 12 ++
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 +++-
src/include/tcop/cmdtaglist.h | 1 +
src/tools/pgindent/typedefs.list | 4 +
11 files changed, 389 insertions(+), 6 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4a4b47ca50..764deef955 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index f24f048054..87b1ea0ab3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -27,6 +36,51 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -34,6 +88,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -185,3 +240,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..a3e5e10d97
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,20 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0bc9db991e..7eff93eff3 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -33,6 +34,41 @@ typedef enum
CT_DELETE_MISSING,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -40,5 +76,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f42efe12b7..0fedd2eae3 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,7 +465,11 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
+ConflictResolverNames
ConflictType
+ConflictTypeNames
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v1-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v1-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 87043c6e3c2e6a941556f21e0752707431e55891 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v1 1/3] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that iolates a NOT DEFERRABLE unique constraint.
update_differ: updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ conflict can only be detected when track_commit_timestamp is
enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 55 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 187 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 50 ++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 44 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 ++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
25 files changed, 695 insertions(+), 151 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a63cc71efa..a9b6f293ea 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8034,6 +8034,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ce37fa6490 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,61 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index efb29adeb3..30393e6d67 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1360,7 +1360,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..2a2d188b19 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ingored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..f24f048054
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,187 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..af73e09b01 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2821,13 +2836,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3005,13 +3017,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..c6b67c692d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5140,6 +5150,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..02aa4a6f32 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..0472fe2e87 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6597,6 +6597,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..0bc9db991e
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..c64f17211d 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61ad417cde..f42efe12b7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v1-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchapplication/octet-stream; name=v1-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchDownload
From d734f2371707c6df005c2424104b74f8bc7281c6 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 25 Jun 2024 15:45:37 +0530
Subject: [PATCH v1 3/3] Implement conflict resolution for INSERT, UPDATE, and
DELETE operations.
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++++-
src/backend/replication/logical/conflict.c | 190 ++++++++++--
src/backend/replication/logical/worker.c | 333 ++++++++++++++++-----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/t/029_on_error.pl | 7 +
6 files changed, 525 insertions(+), 109 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 87b1ea0ab3..a622a0ea8b 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -82,8 +86,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -122,13 +127,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -136,7 +141,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -172,12 +177,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -191,24 +197,29 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -322,6 +333,71 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ *
+ * XXX: Currently, it only handles the simple case of identical table
+ * structures on both Publisher and subscriber. Need to analyze if more
+ * cases can be supported.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -352,7 +428,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
memset(replaces, false, sizeof(replaces));
values[Anum_pg_conflict_conftype - 1] =
- CStringGetTextDatum(stmt->conflict_type);
+ CStringGetTextDatum(stmt->conflict_type);
if (stmt->isReset)
{
@@ -396,3 +472,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index af73e09b01..6e2fda28ec 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,33 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+ }
+
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2708,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2834,12 +2906,25 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2972,19 +3057,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3006,6 +3093,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3015,26 +3105,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
+
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
+
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3044,27 +3193,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3105,10 +3286,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3124,19 +3311,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 7eff93eff3..aa21886ee7 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -72,9 +74,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
--
2.34.1
On Wed, Jun 26, 2024 at 2:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Jun 25, 2024 at 3:39 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 25, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jun 24, 2024 at 1:47 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1].
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1] if you have any comments on patch001.New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'Does setting up resolvers have any meaning without subscriptions? I am
wondering whether we should allow to set up the resolvers at the
subscription level. One benefit is that users don't need to use a
different DDL to set up resolvers. The first patch gives a conflict
detection option at the subscription level, so it would be symmetrical
to provide a resolver at the subscription level. Yet another benefit
could be that it provides users facility to configure different
resolvers for a set of tables belonging to a particular
publication/node.There can be multiple tables included in a publication with varying
business use-cases and thus may need different resolvers set, even
though they all are part of the same publication.Agreed but this is the reason we are planning to keep resolvers at the
table level. Here, I am asking to set resolvers at the subscription
level rather than at the global level.
Okay, got it. I misunderstood earlier that we want to replace table
level resolvers with subscription ones.
Having global configuration has one benefit that if the user has no
requirement to set different resolvers for different subscriptions or
tables, he may always set one global configuration and be done with
it. OTOH, I also agree with benefits coming with subscription level
configuration.
thanks
Shveta
On Thu, Jun 27, 2024 at 8:44 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
Please find the attached 'patch0003', which implements conflict
resolutions according to the global resolver settings.
Thanks for providing the resolver patch.
Please find new patches attached. Changes:
patch002:
--Fixed CFBot compilation failure where a header file was not included
in meson.build
--Also this is the correct version of patch. Previous email has
attached an older version by mistake.
patch004:
This is a WIP progress which attempts to implement Configuration of
table-level resolvers . It has below changes:
--Alter table SET CONFLICT RESOLVER.
--Alter table RESET CONFLICT RESOLVER. <Note that these 2 commands
also take care of resolvers inheritance for partition tables as
discussed in [1]/messages/by-id/CAJpy0uAqegGDbuJk3Z-ku8wYFZyPv7C1KmHCkJ3885O+j5enFg@mail.gmail.com>.
--Resolver inheritance support during 'Alter table ATTACH PARTITION'.
--Resolver inheritance removal during 'Alter table DETACH PARTITION'.
Pending:
--Resolver Inheritance support during 'CREATE TABLE .. PARTITION OF
..'.
--Using tabel-level resolver while resolving conflicts. (Resolver
patch003 still relies on global resolvers).
Please refer [1]/messages/by-id/CAJpy0uAqegGDbuJk3Z-ku8wYFZyPv7C1KmHCkJ3885O+j5enFg@mail.gmail.com for the complete proposal for table-level resolvers.
[1]: /messages/by-id/CAJpy0uAqegGDbuJk3Z-ku8wYFZyPv7C1KmHCkJ3885O+j5enFg@mail.gmail.com
thanks
Shveta
Attachments:
v1-0004-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v1-0004-Configure-table-level-conflict-resolvers.patchDownload
From 52f4e91368bc3ae8bc3267617322ae1407525f48 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Thu, 27 Jun 2024 15:39:51 +0530
Subject: [PATCH v1 4/4] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 67 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 59 ++++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 13 +
.../regress/expected/conflict_resolver.out | 196 ++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 126 ++++++-
14 files changed, 845 insertions(+), 22 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 7b536ac6fd..bd0d48ec78 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 66cda26a25..9841d4e15d 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -4550,6 +4556,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5123,6 +5131,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5535,6 +5552,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6535,6 +6560,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15656,6 +15685,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16244,6 +16280,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20772,3 +20812,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 764deef955..84e1fe0bcd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index a622a0ea8b..927eebf8cc 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -257,9 +262,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -417,10 +422,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -448,7 +452,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -534,3 +538,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..6cba8eaff9
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,59 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* Oid */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation to
+ * configure conflict
+ * resolvers */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index aa21886ee7..296f321865 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -90,4 +90,17 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
TimestampTz committs);
extern bool CanCreateFullTuple(Relation localrel,
LogicalRepTupleData *newtup);
+
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 15c8c0cdba..7f069080f3 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,3 +1,6 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
conftype | confres
@@ -8,20 +11,16 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(4 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
-SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
-SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
ERROR: bbbbb is not a valid conflict resolver
-SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
-RESET CONFLICT RESOLVER for 'ct'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
@@ -48,3 +47,184 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(4 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 6 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers).
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and its child, expect resolver for 'delete_missing'
+--changed and expect 2 new entries added for 'update_differ'.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited entries
+--for ptntable_2 and corresponding entries for ptntable_2_1.
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable;
+-- Test for ALTER TABLE..ATTACH PARTITION
+CREATE TABLE ptntable (c text, a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+----------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+(2 rows)
+
+--Create another paritioned table with one partition
+CREATE TABLE ptntable_1 (b text, c text DEFAULT 'sub1_tab1', a int NOT NULL) PARTITION BY LIST (a);
+CREATE TABLE ptntable_1_1 PARTITION OF ptntable_1 FOR VALUES IN (4, 6);
+--SET resolvers for newly created table ptntable_1
+ALTER TABLE ptntable_1 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'update_differ';
+--Expect total 6 entries, ptntable_1_1 having both entries marked as inherited.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | skip | f
+ ptntable_1 | update_differ | keep_local | f
+ ptntable_1_1 | delete_missing | skip | t
+ ptntable_1_1 | update_differ | keep_local | t
+(6 rows)
+
+--Attach ptntable_1 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_1 FOR VALUES IN (1, 2, 3);
+--Expect 'insert_exists' inherited by both ptntable_1 and ptntable_1_1.
+--The existing entries for 'delete_missing' should still remain same for
+--both ptntable_1 and ptntable_1_1
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | skip | f
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_1 | update_differ | keep_local | f
+ ptntable_1_1 | delete_missing | skip | t
+ ptntable_1_1 | insert_exists | keep_local | t
+ ptntable_1_1 | update_differ | keep_local | t
+(8 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_1;
+--All resolvers of ptntable_1 should be markes as non-inherited.
+--While ptntable_1's resolvers whould still be marked as inherited.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | skip | f
+ ptntable_1 | insert_exists | keep_local | f
+ ptntable_1 | update_differ | keep_local | f
+ ptntable_1_1 | delete_missing | skip | t
+ ptntable_1_1 | insert_exists | keep_local | t
+ ptntable_1_1 | update_differ | keep_local | t
+(8 rows)
+
+DROP TABLE ptntable_1;
+DROP TABLE ptntable;
+--Expect no resolvers.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index e3540d7ac8..6df2802cdb 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,17 +1,19 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
@@ -25,3 +27,119 @@ RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
select * from pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 6 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers).
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and its child, expect resolver for 'delete_missing'
+--changed and expect 2 new entries added for 'update_differ'.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited entries
+--for ptntable_2 and corresponding entries for ptntable_2_1.
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION
+
+CREATE TABLE ptntable (c text, a int PRIMARY KEY, b text) PARTITION BY LIST (a);
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Create another paritioned table with one partition
+CREATE TABLE ptntable_1 (b text, c text DEFAULT 'sub1_tab1', a int NOT NULL) PARTITION BY LIST (a);
+CREATE TABLE ptntable_1_1 PARTITION OF ptntable_1 FOR VALUES IN (4, 6);
+
+--SET resolvers for newly created table ptntable_1
+ALTER TABLE ptntable_1 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'update_differ';
+
+--Expect total 6 entries, ptntable_1_1 having both entries marked as inherited.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_1 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_1 FOR VALUES IN (1, 2, 3);
+
+--Expect 'insert_exists' inherited by both ptntable_1 and ptntable_1_1.
+--The existing entries for 'delete_missing' should still remain same for
+--both ptntable_1 and ptntable_1_1
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_1;
+
+--All resolvers of ptntable_1 should be markes as non-inherited.
+--While ptntable_1's resolvers whould still be marked as inherited.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_1;
+DROP TABLE ptntable;
+
+--Expect no resolvers.
+select relname, confrtype, confrres, confrinherited from pg_class pc,
+ (select confrrelid, confrtype, confrres, confrinherited from pg_conflict_rel)conf
+ where conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v1-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From bdf427d111ed926f70130991527ad3ec9cbc34fd Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Mon, 24 Jun 2024 09:55:15 +0530
Subject: [PATCH v1 2/4] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 215 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 20 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 +++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 50 ++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 27 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 467 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4a4b47ca50..764deef955 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index f24f048054..87b1ea0ab3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -27,6 +36,51 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -34,6 +88,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -185,3 +240,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..a3e5e10d97
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,20 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0bc9db991e..7eff93eff3 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -33,6 +34,41 @@ typedef enum
CT_DELETE_MISSING,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -40,5 +76,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..15c8c0cdba
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,50 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(4 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 969ced994f..4a4a1d9635 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..e3540d7ac8
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,27 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f42efe12b7..85a808882c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v1-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchapplication/octet-stream; name=v1-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchDownload
From 7780d27210c3211eb33fafd66c44a2a9f3f91368 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 25 Jun 2024 15:45:37 +0530
Subject: [PATCH v1 3/4] Implement conflict resolution for INSERT, UPDATE, and
DELETE operations.
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++++-
src/backend/replication/logical/conflict.c | 190 ++++++++++--
src/backend/replication/logical/worker.c | 333 ++++++++++++++++-----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/t/029_on_error.pl | 7 +
6 files changed, 525 insertions(+), 109 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 87b1ea0ab3..a622a0ea8b 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -82,8 +86,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -122,13 +127,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -136,7 +141,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -172,12 +177,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -191,24 +197,29 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -322,6 +333,71 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ *
+ * XXX: Currently, it only handles the simple case of identical table
+ * structures on both Publisher and subscriber. Need to analyze if more
+ * cases can be supported.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -352,7 +428,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
memset(replaces, false, sizeof(replaces));
values[Anum_pg_conflict_conftype - 1] =
- CStringGetTextDatum(stmt->conflict_type);
+ CStringGetTextDatum(stmt->conflict_type);
if (stmt->isReset)
{
@@ -396,3 +472,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index af73e09b01..6e2fda28ec 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,33 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+ }
+
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2708,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2834,12 +2906,25 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2972,19 +3057,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3006,6 +3093,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3015,26 +3105,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
+
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
+
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3044,27 +3193,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3105,10 +3286,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3124,19 +3311,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 7eff93eff3..aa21886ee7 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -72,9 +74,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
--
2.34.1
v1-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v1-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 581efc46960ae54414dc53fc2f8e5b777b7e7e9e Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v1 1/4] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that iolates a NOT DEFERRABLE unique constraint.
update_differ: updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ conflict can only be detected when track_commit_timestamp is
enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 55 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 187 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 50 ++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 44 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 ++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
25 files changed, 695 insertions(+), 151 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a63cc71efa..a9b6f293ea 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8034,6 +8034,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ce37fa6490 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,61 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index efb29adeb3..30393e6d67 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1360,7 +1360,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..2a2d188b19 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ingored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..f24f048054
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,187 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..af73e09b01 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2821,13 +2836,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3005,13 +3017,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..c6b67c692d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5140,6 +5150,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..02aa4a6f32 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..0472fe2e87 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6597,6 +6597,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..0bc9db991e
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..c64f17211d 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61ad417cde..f42efe12b7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
On Thu, Jun 27, 2024 at 4:03 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 27, 2024 at 8:44 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
Please find the attached 'patch0003', which implements conflict
resolutions according to the global resolver settings.Thanks for providing the resolver patch.
Please find new patches attached. Changes:
patch002:
--Fixed CFBot compilation failure where a header file was not included
in meson.build
--Also this is the correct version of patch. Previous email has
attached an older version by mistake.patch004:
This is a WIP progress which attempts to implement Configuration of
table-level resolvers . It has below changes:
--Alter table SET CONFLICT RESOLVER.
--Alter table RESET CONFLICT RESOLVER. <Note that these 2 commands
also take care of resolvers inheritance for partition tables as
discussed in [1]>.
--Resolver inheritance support during 'Alter table ATTACH PARTITION'.
--Resolver inheritance removal during 'Alter table DETACH PARTITION'.Pending:
--Resolver Inheritance support during 'CREATE TABLE .. PARTITION OF
..'.
--Using tabel-level resolver while resolving conflicts. (Resolver
patch003 still relies on global resolvers).Please refer [1] for the complete proposal for table-level resolvers.
Please find v2 attached. Changes are in patch004 only, which are:
--Resolver Inheritance support during 'CREATE TABLE .. PARTITION OF'.
--SPLIT and MERGE partition review and testing (it was missed earlier).
--Test Cases added for all above cases.
thanks
Shveta
Attachments:
v2-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v2-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From bdf427d111ed926f70130991527ad3ec9cbc34fd Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Mon, 24 Jun 2024 09:55:15 +0530
Subject: [PATCH v2 2/4] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 215 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 20 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 +++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 50 ++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 27 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 467 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4a4b47ca50..764deef955 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index f24f048054..87b1ea0ab3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -27,6 +36,51 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -34,6 +88,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -185,3 +240,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..a3e5e10d97
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,20 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0bc9db991e..7eff93eff3 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -33,6 +34,41 @@ typedef enum
CT_DELETE_MISSING,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -40,5 +76,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..15c8c0cdba
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,50 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(4 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 969ced994f..4a4a1d9635 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..e3540d7ac8
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,27 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index f42efe12b7..85a808882c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v2-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchapplication/octet-stream; name=v2-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchDownload
From 7780d27210c3211eb33fafd66c44a2a9f3f91368 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 25 Jun 2024 15:45:37 +0530
Subject: [PATCH v2 3/4] Implement conflict resolution for INSERT, UPDATE, and
DELETE operations.
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++++-
src/backend/replication/logical/conflict.c | 190 ++++++++++--
src/backend/replication/logical/worker.c | 333 ++++++++++++++++-----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/t/029_on_error.pl | 7 +
6 files changed, 525 insertions(+), 109 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 87b1ea0ab3..a622a0ea8b 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -82,8 +86,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -122,13 +127,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -136,7 +141,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -172,12 +177,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -191,24 +197,29 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -322,6 +333,71 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ *
+ * XXX: Currently, it only handles the simple case of identical table
+ * structures on both Publisher and subscriber. Need to analyze if more
+ * cases can be supported.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -352,7 +428,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
memset(replaces, false, sizeof(replaces));
values[Anum_pg_conflict_conftype - 1] =
- CStringGetTextDatum(stmt->conflict_type);
+ CStringGetTextDatum(stmt->conflict_type);
if (stmt->isReset)
{
@@ -396,3 +472,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index af73e09b01..6e2fda28ec 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,33 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+ }
+
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2708,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2834,12 +2906,25 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2972,19 +3057,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3006,6 +3093,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3015,26 +3105,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
+
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
+
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3044,27 +3193,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3105,10 +3286,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3124,19 +3311,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 7eff93eff3..aa21886ee7 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -72,9 +74,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
--
2.34.1
v2-0004-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v2-0004-Configure-table-level-conflict-resolvers.patchDownload
From cc90f43d8ec0f20812327ace9e4b0274102b43c9 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Thu, 27 Jun 2024 15:39:51 +0530
Subject: [PATCH v2 4/4] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 70 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 58 +++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 13 +
.../regress/expected/conflict_resolver.out | 253 +++++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 161 ++++++++-
14 files changed, 933 insertions(+), 28 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 7b536ac6fd..bd0d48ec78 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 66cda26a25..20b746fc80 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -1245,6 +1251,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
CloneForeignKeyConstraints(NULL, parent, rel);
+ /* Inherit conflict resolvers configuration from parent. */
+ InheritTableConflictResolvers(rel, parent);
+
table_close(parent, NoLock);
}
@@ -4550,6 +4559,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5123,6 +5134,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5535,6 +5555,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6535,6 +6563,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15656,6 +15688,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16244,6 +16283,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20772,3 +20815,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 764deef955..84e1fe0bcd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index a622a0ea8b..927eebf8cc 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -257,9 +262,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -417,10 +422,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -448,7 +452,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -534,3 +538,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..636398baf1
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,58 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation
+ * having resolver */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index aa21886ee7..296f321865 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -90,4 +90,17 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
TimestampTz committs);
extern bool CanCreateFullTuple(Relation localrel,
LogicalRepTupleData *newtup);
+
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 15c8c0cdba..8a64aec0f9 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,5 +1,8 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+---------------
delete_missing | skip
@@ -8,26 +11,22 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(4 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
-SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
-SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
ERROR: bbbbb is not a valid conflict resolver
-SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
-RESET CONFLICT RESOLVER for 'ct'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_missing | error
@@ -39,7 +38,7 @@ select * from pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_missing | skip
@@ -48,3 +47,235 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(4 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable_2;
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1_1 | delete_missing | error | t
+ ptntable_1_1 | insert_exists | remote_apply | t
+ ptntable_1_10 | delete_missing | error | t
+ ptntable_1_10 | insert_exists | remote_apply | t
+ ptntable_1_20 | delete_missing | error | t
+ ptntable_1_20 | insert_exists | remote_apply | t
+(8 rows)
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(8 rows)
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index e3540d7ac8..346590303f 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,27 +1,174 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
v2-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v2-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 581efc46960ae54414dc53fc2f8e5b777b7e7e9e Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v2 1/4] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that iolates a NOT DEFERRABLE unique constraint.
update_differ: updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ conflict can only be detected when track_commit_timestamp is
enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 55 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 187 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 50 ++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 44 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 ++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
25 files changed, 695 insertions(+), 151 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index a63cc71efa..a9b6f293ea 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8034,6 +8034,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ce37fa6490 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,61 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index efb29adeb3..30393e6d67 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1360,7 +1360,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..2a2d188b19 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ingored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..f24f048054
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,187 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b5a80fe3e8..af73e09b01 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2821,13 +2836,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3005,13 +3017,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index e324070828..c6b67c692d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4739,6 +4739,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4811,11 +4812,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4854,6 +4861,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4900,6 +4908,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5140,6 +5150,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 865823868f..02aa4a6f32 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f67bf0b892..0472fe2e87 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6529,7 +6529,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6597,6 +6597,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..0bc9db991e
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..c64f17211d 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 61ad417cde..f42efe12b7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -465,6 +465,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
Hi,
On Thu, May 23, 2024 at 3:37 PM shveta malik <shveta.malik@gmail.com> wrote:
DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.
IIUC the 'delete_missing' conflict doesn't cover the case where an
incoming delete message is trying to delete a row that has already
been updated locally or by another node. I think in update/delete
conflict situations, we need to resolve the conflicts based on commit
timestamps like we do for update/update and insert/update conflicts.
For example, suppose there are two node-A and node-B and setup
bi-directional replication, and suppose further that both have the row
with id = 1, consider the following sequences:
09:00:00 DELETE ... WHERE id = 1 on node-A.
09:00:05 UPDATE ... WHERE id = 1 on node-B.
09:00:10 node-A received the update message from node-B.
09:00:15 node-B received the delete message from node-A.
At 09:00:10 on node-A, an update_deleted conflict is generated since
the row on node-A is already deleted locally. Suppose that we use
'apply_or_skip' resolution for this conflict, we convert the update
message into an insertion, so node-A now has the row with id = 1. At
09:00:15 on node-B, the incoming delete message is applied and deletes
the row with id = 1, even though the row has already been modified
locally. The node-A and node-B are now inconsistent. This
inconsistency can be avoided by using 'skip' resolution for the
'update_deleted' conflict on node-A, and 'skip' resolution is the
default method for that actually. However, if we handle it as
'update_missing', the 'apply_or_skip' resolution is used by default.
IIUC with the proposed architecture, DELETE always takes precedence
over UPDATE since both 'update_deleted' and 'update_missing' don't use
commit timestamps to resolve the conflicts. As long as that is true, I
think there is no use case for 'apply_or_skip' and 'apply_or_error'
resolutions in update/delete conflict cases. In short, I think we need
something like 'delete_differ' conflict type as well. FYI PGD and
Oracle GoldenGate seem to have this conflict type[1]https://www.enterprisedb.com/docs/pgd/latest/consistency/conflicts/#updatedelete-conflicts[2]https://docs.oracle.com/goldengate/c1230/gg-winux/GWUAD/configuring-conflict-detection-and-resolution.htm (see DELETEROWEXISTS conflict type).
The 'delete'_differ' conflict type would have at least
'latest_timestamp_wins' resolution. With the timestamp based
resolution method, we would deal with update/delete conflicts as
follows:
09:00:00: DELETE ... WHERE id = 1 on node-A.
09:00:05: UPDATE ... WHERE id = 1 on node-B.
- the updated row doesn't have the origin since it's a local change.
09:00:10: node-A received the update message from node-B.
- the incoming update message has the origin of node-B whereas the
local row is already removed locally.
- 'update_deleted' conflict is generated.
- do the insert of the new row instead, because the commit
timestamp of UPDATE is newer than DELETE's one.
09:00:15: node-B received the delete message from node-A.
- the incoming delete message has the origin of node-B whereas the
(updated) row doesn't have the origin.
- 'update_differ' conflict is generated.
- discard DELETE, because the commit timestamp of UPDATE is newer
than DELETE' one.ard DELETE, because the commit timestamp of UPDATE is
newer than DELETE' one.
As a result, both nodes have the new version row.
Regards,
[1]: https://www.enterprisedb.com/docs/pgd/latest/consistency/conflicts/#updatedelete-conflicts
[2]: https://docs.oracle.com/goldengate/c1230/gg-winux/GWUAD/configuring-conflict-detection-and-resolution.htm (see DELETEROWEXISTS conflict type)
(see DELETEROWEXISTS conflict type)
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
On Thu, Jun 27, 2024 at 1:14 PM Nisha Moond <nisha.moond412@gmail.com>
wrote:
Please find the attached 'patch0003', which implements conflict
resolutions according to the global resolver settings.Summary of Conflict Resolutions Implemented in 'patch0003':
INSERT Conflicts:
------------------------
1) Conflict Type: 'insert_exists'Supported Resolutions:
a) 'remote_apply': Convert the INSERT to an UPDATE and apply.
b) 'keep_local': Ignore the incoming (conflicting) INSERT and retain
the local tuple.
c) 'error': The apply worker will error out and restart.
Hi Nisha,
While testing the patch, when conflict resolution is configured and
insert_exists is set to "remote_apply", I see this warning in the logs due
to a resource not being closed:
2024-07-01 02:52:59.427 EDT [20304] LOG: conflict insert_exists detected
on relation "public.test1"
2024-07-01 02:52:59.427 EDT [20304] DETAIL: Key already exists. Applying
resolution method "remote_apply"
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for
replication origin "pg_16417" during message type "INSERT" for replication
target relation "public.test1" in transaction 763, finished at 0/15E7F68
2024-07-01 02:52:59.427 EDT [20304] WARNING: resource was not closed:
[138]: (rel=base/5/16413, blockNum=0, flags=0x93800000, refcount=1 1) 2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for replication origin "pg_16417" during message type "COMMIT" in transaction 763, finished at 0/15E7F68 2024-07-01 02:52:59.427 EDT [20304] WARNING: resource was not closed: TupleDesc 0x7f8c0439e448 (16402,-1) 2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for replication origin "pg_16417" during message type "COMMIT" in transaction 763, finished at 0/15E7F68
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for
replication origin "pg_16417" during message type "COMMIT" in transaction
763, finished at 0/15E7F68
2024-07-01 02:52:59.427 EDT [20304] WARNING: resource was not closed:
TupleDesc 0x7f8c0439e448 (16402,-1)
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for
replication origin "pg_16417" during message type "COMMIT" in transaction
763, finished at 0/15E7F68
regards,
Ajin Cherian
Fujitsu Australia
On Thu, Jun 27, 2024 at 1:50 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jun 26, 2024 at 2:33 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Tue, Jun 25, 2024 at 3:39 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 25, 2024 at 3:12 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jun 24, 2024 at 1:47 PM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Jun 20, 2024 at 6:41 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
In the second patch, we can implement simple built-in resolution
strategies like apply and skip (which can be named as remote_apply and
keep_local, see [3][4] for details on these strategies) with ERROR or
LOG being the default strategy. We can allow these strategies to be
configured at the global and table level.Before we implement resolvers, we need a way to configure them. Please
find the patch002 which attempts to implement Global Level Conflict
Resolvers Configuration. Note that patch002 is dependent upon
Conflict-Detection patch001 which is reviewed in another thread [1].
I have attached patch001 here for convenience and to avoid CFBot
failures. But please use [1] if you have any comments on patch001.New DDL commands in patch002 are:
To set global resolver for given conflcit_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'To reset to default resolver:
RESET CONFLICT RESOLVER FOR 'conflict_type'Does setting up resolvers have any meaning without subscriptions? I am
wondering whether we should allow to set up the resolvers at the
subscription level. One benefit is that users don't need to use a
different DDL to set up resolvers. The first patch gives a conflict
detection option at the subscription level, so it would be symmetrical
to provide a resolver at the subscription level. Yet another benefit
could be that it provides users facility to configure different
resolvers for a set of tables belonging to a particular
publication/node.There can be multiple tables included in a publication with varying
business use-cases and thus may need different resolvers set, even
though they all are part of the same publication.Agreed but this is the reason we are planning to keep resolvers at the
table level. Here, I am asking to set resolvers at the subscription
level rather than at the global level.Okay, got it. I misunderstood earlier that we want to replace table
level resolvers with subscription ones.
Having global configuration has one benefit that if the user has no
requirement to set different resolvers for different subscriptions or
tables, he may always set one global configuration and be done with
it. OTOH, I also agree with benefits coming with subscription level
configuration.
Setting resolvers at table-level and subscription-level sounds good to
me. DDLs for setting resolvers at subscription-level would need the
subscription name to be specified? And another question is: a
table-level resolver setting is precedent over all subscriber-level
resolver settings in the database?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
On Mon, Jul 1, 2024 at 11:47 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
On Thu, May 23, 2024 at 3:37 PM shveta malik <shveta.malik@gmail.com> wrote:
DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.IIUC the 'delete_missing' conflict doesn't cover the case where an
incoming delete message is trying to delete a row that has already
been updated locally or by another node. I think in update/delete
conflict situations, we need to resolve the conflicts based on commit
timestamps like we do for update/update and insert/update conflicts.For example, suppose there are two node-A and node-B and setup
bi-directional replication, and suppose further that both have the row
with id = 1, consider the following sequences:09:00:00 DELETE ... WHERE id = 1 on node-A.
09:00:05 UPDATE ... WHERE id = 1 on node-B.
09:00:10 node-A received the update message from node-B.
09:00:15 node-B received the delete message from node-A.At 09:00:10 on node-A, an update_deleted conflict is generated since
the row on node-A is already deleted locally. Suppose that we use
'apply_or_skip' resolution for this conflict, we convert the update
message into an insertion, so node-A now has the row with id = 1. At
09:00:15 on node-B, the incoming delete message is applied and deletes
the row with id = 1, even though the row has already been modified
locally. The node-A and node-B are now inconsistent. This
inconsistency can be avoided by using 'skip' resolution for the
'update_deleted' conflict on node-A, and 'skip' resolution is the
default method for that actually. However, if we handle it as
'update_missing', the 'apply_or_skip' resolution is used by default.IIUC with the proposed architecture, DELETE always takes precedence
over UPDATE since both 'update_deleted' and 'update_missing' don't use
commit timestamps to resolve the conflicts. As long as that is true, I
think there is no use case for 'apply_or_skip' and 'apply_or_error'
resolutions in update/delete conflict cases. In short, I think we need
something like 'delete_differ' conflict type as well. FYI PGD and
Oracle GoldenGate seem to have this conflict type[1][2].
Your explanation makes sense to me and I agree that we should
implement 'delete_differ' conflict type.
The 'delete'_differ' conflict type would have at least
'latest_timestamp_wins' resolution. With the timestamp based
resolution method, we would deal with update/delete conflicts as
follows:09:00:00: DELETE ... WHERE id = 1 on node-A.
09:00:05: UPDATE ... WHERE id = 1 on node-B.
- the updated row doesn't have the origin since it's a local change.
09:00:10: node-A received the update message from node-B.
- the incoming update message has the origin of node-B whereas the
local row is already removed locally.
- 'update_deleted' conflict is generated.
FYI, as of now, we don't have a reliable way to detect
'update_deleted' type of conflicts but we had some discussion about
the same [1]/messages/by-id/CAA4eK1Lj-PWrP789KnKxZydisHajd38rSihWXO8MVBLDwxG1Kg@mail.gmail.com.
- do the insert of the new row instead, because the commit
timestamp of UPDATE is newer than DELETE's one.
09:00:15: node-B received the delete message from node-A.
- the incoming delete message has the origin of node-B whereas the
(updated) row doesn't have the origin.
- 'update_differ' conflict is generated.
- discard DELETE, because the commit timestamp of UPDATE is newer
than DELETE' one.ard DELETE, because the commit timestamp of UPDATE is
newer than DELETE' one.As a result, both nodes have the new version row.
Right, it seems to me that we should implement 'latest_time_wins' if
we want consistency in such cases.
[1]: /messages/by-id/CAA4eK1Lj-PWrP789KnKxZydisHajd38rSihWXO8MVBLDwxG1Kg@mail.gmail.com
--
With Regards,
Amit Kapila.
On Mon, Jul 1, 2024 at 1:35 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Setting resolvers at table-level and subscription-level sounds good to
me. DDLs for setting resolvers at subscription-level would need the
subscription name to be specified?
Yes, it should be part of the ALTER/CREATE SUBSCRIPTION command. One
idea could be to have syntax as follows:
ALTER SUBSCRIPTION name SET CONFLICT RESOLVER 'conflict_resolver' FOR
'conflict_type';
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR 'conflict_type';
CREATE SUBSCRIPTION subscription_name CONNECTION 'conninfo'
PUBLICATION publication_name [, ...] CONFLICT RESOLVER
'conflict_resolver' FOR 'conflict_type';
And another question is: a
table-level resolver setting is precedent over all subscriber-level
resolver settings in the database?
Yes.
--
With Regards,
Amit Kapila.
On Mon, Jul 1, 2024 at 11:47 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Hi,
On Thu, May 23, 2024 at 3:37 PM shveta malik <shveta.malik@gmail.com> wrote:
DELETE
================
Conflict Type:
----------------
delete_missing: An incoming delete is trying to delete a row on a
target node which does not exist.IIUC the 'delete_missing' conflict doesn't cover the case where an
incoming delete message is trying to delete a row that has already
been updated locally or by another node. I think in update/delete
conflict situations, we need to resolve the conflicts based on commit
timestamps like we do for update/update and insert/update conflicts.For example, suppose there are two node-A and node-B and setup
bi-directional replication, and suppose further that both have the row
with id = 1, consider the following sequences:09:00:00 DELETE ... WHERE id = 1 on node-A.
09:00:05 UPDATE ... WHERE id = 1 on node-B.
09:00:10 node-A received the update message from node-B.
09:00:15 node-B received the delete message from node-A.At 09:00:10 on node-A, an update_deleted conflict is generated since
the row on node-A is already deleted locally. Suppose that we use
'apply_or_skip' resolution for this conflict, we convert the update
message into an insertion, so node-A now has the row with id = 1. At
09:00:15 on node-B, the incoming delete message is applied and deletes
the row with id = 1, even though the row has already been modified
locally. The node-A and node-B are now inconsistent. This
inconsistency can be avoided by using 'skip' resolution for the
'update_deleted' conflict on node-A, and 'skip' resolution is the
default method for that actually. However, if we handle it as
'update_missing', the 'apply_or_skip' resolution is used by default.IIUC with the proposed architecture, DELETE always takes precedence
over UPDATE since both 'update_deleted' and 'update_missing' don't use
commit timestamps to resolve the conflicts. As long as that is true, I
think there is no use case for 'apply_or_skip' and 'apply_or_error'
resolutions in update/delete conflict cases. In short, I think we need
something like 'delete_differ' conflict type as well.
Thanks for the feedback. Sure, we can have 'delete_differ'.
FYI PGD and
Oracle GoldenGate seem to have this conflict type[1][2].The 'delete'_differ' conflict type would have at least
'latest_timestamp_wins' resolution. With the timestamp based
resolution method, we would deal with update/delete conflicts as
follows:09:00:00: DELETE ... WHERE id = 1 on node-A.
09:00:05: UPDATE ... WHERE id = 1 on node-B.
- the updated row doesn't have the origin since it's a local change.
09:00:10: node-A received the update message from node-B.
- the incoming update message has the origin of node-B whereas the
local row is already removed locally.
- 'update_deleted' conflict is generated.
- do the insert of the new row instead, because the commit
timestamp of UPDATE is newer than DELETE's one.
So, are you suggesting to support latest_tmestamp_wins for
'update_deleted' case? And shall 'latest_tmestamp_wins' be default
then instead of 'skip'? In some cases, the complete row can not be
constructed, and then 'insertion' might not be possible even if the
timestamp of 'update' is latest. Then shall we skip or error out at
latest_tmestamp_wins config?
Even if we support 'latest_timestamp_wins' as default, we can still
have 'apply_or_skip' and 'apply_or_error' as other options for
'update_deleted' case. Or do you suggest getting rid of these options
completely?
09:00:15: node-B received the delete message from node-A.
- the incoming delete message has the origin of node-B whereas the
(updated) row doesn't have the origin.
- 'update_differ' conflict is generated.
Here, do you mean 'delete_differ' conflict is generated?
thanks
Shveta
On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
node which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2 and the timestamp will also show the same at any other
node if they receive these 2 changes.The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame. Generally, the ideal configuration for
max_clock_skew should be in multiple of the network round trip time.
Assuming this configuration, we wouldn’t encounter this problem
because for change-2 to be caused by change-1, the client would need
to get confirmation of change-1 and then trigger change-2, which would
take at least 2-3 network round trips.
As we agreed, the subscriber should wait before applying an operation
if the commit timestamp of the currently replayed transaction is in
the future and the difference exceeds the maximum clock skew. This
raises the question: should the subscriber wait only for insert,
update, and delete operations when timestamp-based resolution methods
are set, or should it wait regardless of the type of remote operation,
the presence or absence of conflicts, and the resolvers configured?
I believe the latter approach is the way to go i.e. this should be
independent of CDR, though needed by CDR for better timestamp based
resolutions. Thoughts?
thanks
Shveta
On Tue, Jul 2, 2024 at 2:40 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
node which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2 and the timestamp will also show the same at any other
node if they receive these 2 changes.The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame. Generally, the ideal configuration for
max_clock_skew should be in multiple of the network round trip time.
Assuming this configuration, we wouldn’t encounter this problem
because for change-2 to be caused by change-1, the client would need
to get confirmation of change-1 and then trigger change-2, which would
take at least 2-3 network round trips.As we agreed, the subscriber should wait before applying an operation
if the commit timestamp of the currently replayed transaction is in
the future and the difference exceeds the maximum clock skew. This
raises the question: should the subscriber wait only for insert,
update, and delete operations when timestamp-based resolution methods
are set, or should it wait regardless of the type of remote operation,
the presence or absence of conflicts, and the resolvers configured?
I believe the latter approach is the way to go i.e. this should be
independent of CDR, though needed by CDR for better timestamp based
resolutions. Thoughts?
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR. IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 10:47 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jul 2, 2024 at 2:40 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jun 19, 2024 at 1:52 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Tue, Jun 18, 2024 at 3:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jun 18, 2024 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
I tried to work out a few scenarios with this, where the apply worker
will wait until its local clock hits 'remote_commit_tts - max_skew
permitted'. Please have a look.Let's say, we have a GUC to configure max_clock_skew permitted.
Resolver is last_update_wins in both cases.
----------------
1) Case 1: max_clock_skew set to 0 i.e. no tolerance for clock skew.Remote Update with commit_timestamp = 10.20AM.
Local clock (which is say 5 min behind) shows = 10.15AM.When remote update arrives at local node, we see that skew is greater
than max_clock_skew and thus apply worker waits till local clock hits
'remote's commit_tts - max_clock_skew' i.e. till 10.20 AM. Once the
local clock hits 10.20 AM, the worker applies the remote change with
commit_tts of 10.20AM. In the meantime (during wait period of apply
worker)) if some local update on same row has happened at say 10.18am,
that will applied first, which will be later overwritten by above
remote change of 10.20AM as remote-change's timestamp appear more
latest, even though it has happened earlier than local change.For the sake of simplicity let's call the change that happened at
10:20 AM change-1 and the change that happened at 10:15 as change-2
and assume we are talking about the synchronous commit only.I think now from an application perspective the change-1 wouldn't have
caused the change-2 because we delayed applying change-2 on the local
node which would have delayed the confirmation of the change-1 to the
application that means we have got the change-2 on the local node
without the confirmation of change-1 hence change-2 has no causal
dependency on the change-1. So it's fine that we perform change-1
before change-2 and the timestamp will also show the same at any other
node if they receive these 2 changes.The goal is to ensure that if we define the order where change-2
happens before change-1, this same order should be visible on all
other nodes. This will hold true because the commit timestamp of
change-2 is earlier than that of change-1.2) Case 2: max_clock_skew is set to 2min.
Remote Update with commit_timestamp=10.20AM
Local clock (which is say 5 min behind) = 10.15AM.Now apply worker will notice skew greater than 2min and thus will wait
till local clock hits 'remote's commit_tts - max_clock_skew' i.e.
10.18 and will apply the change with commit_tts of 10.20 ( as we
always save the origin's commit timestamp into local commit_tts, see
RecordTransactionCommit->TransactionTreeSetCommitTsData). Now lets say
another local update is triggered at 10.19am, it will be applied
locally but it will be ignored on remote node. On the remote node ,
the existing change with a timestamp of 10.20 am will win resulting in
data divergence.Let's call the 10:20 AM change as a change-1 and the change that
happened at 10:19 as change-2IIUC, although we apply the change-1 at 10:18 AM the commit_ts of that
commit_ts of that change is 10:20, and the same will be visible to all
other nodes. So in conflict resolution still the change-1 happened
after the change-2 because change-2's commit_ts is 10:19 AM. Now
there could be a problem with the causal order because we applied the
change-1 at 10:18 AM so the application might have gotten confirmation
at 10:18 AM and the change-2 of the local node may be triggered as a
result of confirmation of the change-1 that means now change-2 has a
causal dependency on the change-1 but commit_ts shows change-2
happened before the change-1 on all the nodes.So, is this acceptable? I think yes because the user has configured a
maximum clock skew of 2 minutes, which means the detected order might
not always align with the causal order for transactions occurring
within that time frame. Generally, the ideal configuration for
max_clock_skew should be in multiple of the network round trip time.
Assuming this configuration, we wouldn’t encounter this problem
because for change-2 to be caused by change-1, the client would need
to get confirmation of change-1 and then trigger change-2, which would
take at least 2-3 network round trips.As we agreed, the subscriber should wait before applying an operation
if the commit timestamp of the currently replayed transaction is in
the future and the difference exceeds the maximum clock skew. This
raises the question: should the subscriber wait only for insert,
update, and delete operations when timestamp-based resolution methods
are set, or should it wait regardless of the type of remote operation,
the presence or absence of conflicts, and the resolvers configured?
I believe the latter approach is the way to go i.e. this should be
independent of CDR, though needed by CDR for better timestamp based
resolutions. Thoughts?Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.
+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?
+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?
thanks
Shveta
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?
But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no? I mean we should wait before committing because we are
considering this remote transaction to be in the future and we do not
want to confirm the commit of this transaction to the remote node
before the local clock reaches the record commit_ts to preserve the
causal order. However, we can still perform conflict resolution
beforehand since we already know the commit_ts. The conflict
resolution function will be something like "out_version =
CRF(version1_commit_ts, version2_commit_ts)," so the result should be
the same regardless of when we apply it, correct? From a performance
standpoint, wouldn't it be beneficial to perform as much work as
possible in advance? By the time we apply all the operations, the
local clock might already be in sync with the commit_ts of the remote
transaction. Am I missing something?
However, while thinking about this, I'm wondering about how we will
handle the streaming of in-progress transactions. If we start applying
with parallel workers, we might not know the commit_ts of those
transactions since they may not have been committed yet. One simple
option could be to prevent parallel workers from applying in-progress
transactions when CDR is set up. Instead, we could let these
transactions spill to files and only apply them once we receive the
commit record.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no? I mean we should wait before committing because we are
considering this remote transaction to be in the future and we do not
want to confirm the commit of this transaction to the remote node
before the local clock reaches the record commit_ts to preserve the
causal order. However, we can still perform conflict resolution
beforehand since we already know the commit_ts. The conflict
resolution function will be something like "out_version =
CRF(version1_commit_ts, version2_commit_ts)," so the result should be
the same regardless of when we apply it, correct? From a performance
standpoint, wouldn't it be beneficial to perform as much work as
possible in advance? By the time we apply all the operations, the
local clock might already be in sync with the commit_ts of the remote
transaction. Am I missing something?
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.
However, while thinking about this, I'm wondering about how we will
handle the streaming of in-progress transactions. If we start applying
with parallel workers, we might not know the commit_ts of those
transactions since they may not have been committed yet. One simple
option could be to prevent parallel workers from applying in-progress
transactions when CDR is set up. Instead, we could let these
transactions spill to files and only apply them once we receive the
commit record.
Agreed, we should do it as you have suggested and document it.
--
With Regards,
Amit Kapila.
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.
But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?
Also, we already have a handling of parallel apply workers so if we do
not have an issue of deadlock there or if we can handle those issues
there we can do it here as well no?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?
That would be true even if we wait just before applying the commit
record considering the transaction is small and the wait time is
large.
Also, we already have a handling of parallel apply workers so if we do
not have an issue of deadlock there or if we can handle those issues
there we can do it here as well no?
Parallel apply workers won't wait for a long time. There is some
similarity and in both cases, deadlock will be detected but chances of
such implementation-related deadlocks will be higher if we start
waiting for a random amount of times. The other possibility is that we
can keep a cap on the max clock skew time above which we will give
ERROR even if the user has configured wait. This is because anyway the
system will be choked (walsender won't be able to send more data,
vacuum on publisher won't be able to remove dead rows) if we wait for
longer times. But even with that, I am not sure if waiting after
holding locks is a good idea or gives us the benefit that is worth the
risk of deadlocks.
--
With Regards,
Amit Kapila.
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no?
I would like to highlight one point here that the resultant data may
be different depending upon at what stage (begin or commit) we
conclude to wait. Example:
--max_clock_skew set to 0 i.e. no tolerance for clock skew.
--Remote Update with commit_timestamp = 10.20AM.
--Local clock (which is say 5 min behind) shows = 10.15AM.
Case 1: Wait during Begin:
When remote update arrives at local node, apply worker waits till
local clock hits 'remote's commit_tts - max_clock_skew' i.e. till
10.20 AM. In the meantime (during the wait period of apply worker) if
some local update on the same row has happened at say 10.18am (local
clock), that will be applied first. Now when apply worker's wait is
over, it will detect 'update_diffe'r conflict and as per
'last_update_win', remote_tuple will win as 10.20 is latest than
10.18.
Case 2: Wait during Commit:
When remote update arrives at local node, it finds no conflict and
goes for commit. But before commit, it waits till the local clock hits
10.20 AM. In the meantime (during wait period of apply worker)) if
some local update is trying to update the same row say at 10.18, it
has to wait (due to locks taken by remote update on that row) and
remote tuple will get committed first with commit timestamp of 10.20.
Then local update will proceed and will overwrite remote tuple.
So in case1, remote tuple is the final change while in case2, local
tuple is the final change.
thanks
Shveta
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?That would be true even if we wait just before applying the commit
record considering the transaction is small and the wait time is
large.
What I am saying is that if we are not applying the whole transaction,
it means we are not receiving it either unless we plan to spill it to
a file. If we don't spill it to a file, the network buffer will fill
up very quickly. This issue wouldn't occur if we waited right before
the commit because, by that time, we would have already received all
the data from the network.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?That would be true even if we wait just before applying the commit
record considering the transaction is small and the wait time is
large.Also, we already have a handling of parallel apply workers so if we do
not have an issue of deadlock there or if we can handle those issues
there we can do it here as well no?Parallel apply workers won't wait for a long time. There is some
similarity and in both cases, deadlock will be detected but chances of
such implementation-related deadlocks will be higher if we start
waiting for a random amount of times. The other possibility is that we
can keep a cap on the max clock skew time above which we will give
ERROR even if the user has configured wait.
+1. But I think cap has to be on wait-time. As an example, let's say
the user has configured 'clock skew tolerance' of 10sec while the
actual clock skew between nodes is 5 min. It means, we will mostly
have to wait '5 min - 10sec' to bring the clock skew to a tolerable
limit, which is a huge waiting time. We can keep a max limit on this
wait time.
thanks
Shveta
On Wed, Jul 3, 2024 at 4:02 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no?I would like to highlight one point here that the resultant data may
be different depending upon at what stage (begin or commit) we
conclude to wait. Example:--max_clock_skew set to 0 i.e. no tolerance for clock skew.
--Remote Update with commit_timestamp = 10.20AM.
--Local clock (which is say 5 min behind) shows = 10.15AM.Case 1: Wait during Begin:
When remote update arrives at local node, apply worker waits till
local clock hits 'remote's commit_tts - max_clock_skew' i.e. till
10.20 AM. In the meantime (during the wait period of apply worker) if
some local update on the same row has happened at say 10.18am (local
clock), that will be applied first. Now when apply worker's wait is
over, it will detect 'update_diffe'r conflict and as per
'last_update_win', remote_tuple will win as 10.20 is latest than
10.18.Case 2: Wait during Commit:
When remote update arrives at local node, it finds no conflict and
goes for commit. But before commit, it waits till the local clock hits
10.20 AM. In the meantime (during wait period of apply worker)) if
some local update is trying to update the same row say at 10.18, it
has to wait (due to locks taken by remote update on that row) and
remote tuple will get committed first with commit timestamp of 10.20.
Then local update will proceed and will overwrite remote tuple.So in case1, remote tuple is the final change while in case2, local
tuple is the final change.
Got it, but which case is correct, I think both. Because in case-1
local commit's commit_ts is 10:18 and the remote commit's commit_ts is
10:20 so remote apply wins. And case 2, the remote commit's commit_ts
is 10:20 whereas the local commit's commit_ts must be 10:20 + delta
(because it waited for the remote transaction to get committed).
Now say which is better, in case-1 we have to make the remote apply to
wait at the beginning state without knowing what would be the local
clock when it actually comes to commit, it may so happen that if we
choose case-2 by the time the remote transaction finish applying the
local clock is beyond 10:20 and we do not even need to wait?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?That would be true even if we wait just before applying the commit
record considering the transaction is small and the wait time is
large.What I am saying is that if we are not applying the whole transaction,
it means we are not receiving it either unless we plan to spill it to
a file. If we don't spill it to a file, the network buffer will fill
up very quickly. This issue wouldn't occur if we waited right before
the commit because, by that time, we would have already received all
the data from the network.
We would have received the transaction data but there could be other
transactions that need to wait because the apply worker is waiting
before the commit. So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.
--
With Regards,
Amit Kapila.
On Wed, Jul 3, 2024 at 4:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 3:35 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 2:16 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.But if we make it wait at the very first operation that means we will
not suck more decoded data from the network and wouldn't that make the
sender wait for the network buffer to get sucked in by the receiver?That would be true even if we wait just before applying the commit
record considering the transaction is small and the wait time is
large.What I am saying is that if we are not applying the whole transaction,
it means we are not receiving it either unless we plan to spill it to
a file. If we don't spill it to a file, the network buffer will fill
up very quickly. This issue wouldn't occur if we waited right before
the commit because, by that time, we would have already received all
the data from the network.We would have received the transaction data but there could be other
transactions that need to wait because the apply worker is waiting
before the commit.
Yeah, that's a valid point, can parallel apply worker help here?
So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.
Yes, spilling to file or cap on the wait time should help, and as I
said above maybe a parallel apply worker can also help.
So I agree that the problem with network buffers arises in both cases,
whether we wait before committing or before beginning. So keeping that
in mind I don't have any strong objections against waiting at the
beginning if it simplifies the design compared to waiting at the
commit.
However, one point to remember in favor of waiting before applying the
commit is that if we decide to wait before beginning the transaction,
we would end up waiting in many more cases compared to waiting before
committing. Because in cases, when transactions are large and the
clock skew is small, the local clock would have already passed the
remote commit_ts by the time we reach the commit.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 4:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:02 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no?I would like to highlight one point here that the resultant data may
be different depending upon at what stage (begin or commit) we
conclude to wait. Example:--max_clock_skew set to 0 i.e. no tolerance for clock skew.
--Remote Update with commit_timestamp = 10.20AM.
--Local clock (which is say 5 min behind) shows = 10.15AM.Case 1: Wait during Begin:
When remote update arrives at local node, apply worker waits till
local clock hits 'remote's commit_tts - max_clock_skew' i.e. till
10.20 AM. In the meantime (during the wait period of apply worker) if
some local update on the same row has happened at say 10.18am (local
clock), that will be applied first. Now when apply worker's wait is
over, it will detect 'update_diffe'r conflict and as per
'last_update_win', remote_tuple will win as 10.20 is latest than
10.18.Case 2: Wait during Commit:
When remote update arrives at local node, it finds no conflict and
goes for commit. But before commit, it waits till the local clock hits
10.20 AM. In the meantime (during wait period of apply worker)) if
some local update is trying to update the same row say at 10.18, it
has to wait (due to locks taken by remote update on that row) and
remote tuple will get committed first with commit timestamp of 10.20.
Then local update will proceed and will overwrite remote tuple.So in case1, remote tuple is the final change while in case2, local
tuple is the final change.Got it, but which case is correct, I think both. Because in case-1
local commit's commit_ts is 10:18 and the remote commit's commit_ts is
10:20 so remote apply wins. And case 2, the remote commit's commit_ts
is 10:20 whereas the local commit's commit_ts must be 10:20 + delta
(because it waited for the remote transaction to get committed).Now say which is better, in case-1 we have to make the remote apply to
wait at the beginning state without knowing what would be the local
clock when it actually comes to commit, it may so happen that if we
choose case-2 by the time the remote transaction finish applying the
local clock is beyond 10:20 and we do not even need to wait?
yes, agree that wait time could be lesser to some extent in case 2.
But the wait during commit will make user operations on the same row
wait, without user having any clue on concurrent blocking operations.
I am not sure if it will be acceptable.
thanks
Shveta
On Wed, Jul 3, 2024 at 5:08 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:02 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no?I would like to highlight one point here that the resultant data may
be different depending upon at what stage (begin or commit) we
conclude to wait. Example:--max_clock_skew set to 0 i.e. no tolerance for clock skew.
--Remote Update with commit_timestamp = 10.20AM.
--Local clock (which is say 5 min behind) shows = 10.15AM.Case 1: Wait during Begin:
When remote update arrives at local node, apply worker waits till
local clock hits 'remote's commit_tts - max_clock_skew' i.e. till
10.20 AM. In the meantime (during the wait period of apply worker) if
some local update on the same row has happened at say 10.18am (local
clock), that will be applied first. Now when apply worker's wait is
over, it will detect 'update_diffe'r conflict and as per
'last_update_win', remote_tuple will win as 10.20 is latest than
10.18.Case 2: Wait during Commit:
When remote update arrives at local node, it finds no conflict and
goes for commit. But before commit, it waits till the local clock hits
10.20 AM. In the meantime (during wait period of apply worker)) if
some local update is trying to update the same row say at 10.18, it
has to wait (due to locks taken by remote update on that row) and
remote tuple will get committed first with commit timestamp of 10.20.
Then local update will proceed and will overwrite remote tuple.So in case1, remote tuple is the final change while in case2, local
tuple is the final change.Got it, but which case is correct, I think both. Because in case-1
local commit's commit_ts is 10:18 and the remote commit's commit_ts is
10:20 so remote apply wins. And case 2, the remote commit's commit_ts
is 10:20 whereas the local commit's commit_ts must be 10:20 + delta
(because it waited for the remote transaction to get committed).Now say which is better, in case-1 we have to make the remote apply to
wait at the beginning state without knowing what would be the local
clock when it actually comes to commit, it may so happen that if we
choose case-2 by the time the remote transaction finish applying the
local clock is beyond 10:20 and we do not even need to wait?yes, agree that wait time could be lesser to some extent in case 2.
But the wait during commit will make user operations on the same row
wait, without user having any clue on concurrent blocking operations.
I am not sure if it will be acceptable.
I don't think there is any problem with the acceptance of user
experience because even while applying the remote transaction
(irrespective of whether we implement this wait feature) the user
transaction might have to wait if updating the common rows right?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Wed, Jul 3, 2024 at 12:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:29 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 11:00 AM shveta malik <shveta.malik@gmail.com> wrote:
Yes, I also think it should be independent of CDR. IMHO, it should be
based on the user-configured maximum clock skew tolerance and can be
independent of CDR.+1
IIUC we would make the remote apply wait just
before committing if the remote commit timestamp is ahead of the local
clock by more than the maximum clock skew tolerance, is that correct?+1 on condition to wait.
But I think we should make apply worker wait during begin
(apply_handle_begin) instead of commit. It makes more sense to delay
the entire operation to manage clock-skew rather than the commit
alone. And only then CDR's timestamp based resolution which are much
prior to commit-stage can benefit from this. Thoughts?But do we really need to wait at apply_handle_begin()? I mean if we
already know the commit_ts then we can perform the conflict resolution
no? I mean we should wait before committing because we are
considering this remote transaction to be in the future and we do not
want to confirm the commit of this transaction to the remote node
before the local clock reaches the record commit_ts to preserve the
causal order. However, we can still perform conflict resolution
beforehand since we already know the commit_ts. The conflict
resolution function will be something like "out_version =
CRF(version1_commit_ts, version2_commit_ts)," so the result should be
the same regardless of when we apply it, correct? From a performance
standpoint, wouldn't it be beneficial to perform as much work as
possible in advance? By the time we apply all the operations, the
local clock might already be in sync with the commit_ts of the remote
transaction. Am I missing something?But waiting after applying the operations and before applying the
commit would mean that we need to wait with the locks held. That could
be a recipe for deadlocks in the system. I see your point related to
performance but as we are not expecting clock skew in normal cases, we
shouldn't be too much bothered on the performance due to this. If
there is clock skew, we expect users to fix it, this is just a
worst-case aid for users.
Please find the new patch set. patch004 is the new patch which
attempts to implement:
1) Either wait or error out on clock skew as configured. Please note
that currently wait is implemented during 'begin'. Once the ongoing
discussion is concluded, it can be changed as needed.
2) last_update_wins resolver. Thanks Nisha for providing the resolver
related changes.
Next to be done:
1) parallel apply worker related changes as mentioned in [1]/messages/by-id/CAFiTN-sf23K=sRsnxw-BKNJqg5P6JXcqXBBkx=EULX8QGSQYaw@mail.gmail.com
2) cap on wait time due to clock skew
3) resolvers for delete_differ as conflict detection thread [2]/messages/by-id/OS0PR01MB571686E464A325F26CEFCCEF94DD2@OS0PR01MB5716.jpnprd01.prod.outlook.com has
implemented detection for that.
[1]: /messages/by-id/CAFiTN-sf23K=sRsnxw-BKNJqg5P6JXcqXBBkx=EULX8QGSQYaw@mail.gmail.com
[2]: /messages/by-id/OS0PR01MB571686E464A325F26CEFCCEF94DD2@OS0PR01MB5716.jpnprd01.prod.outlook.com
thanks
Shveta
Attachments:
v3-0005-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v3-0005-Configure-table-level-conflict-resolvers.patchDownload
From a2ac74aea89fec60049f6bd5eb4f18515ac7637f Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Wed, 3 Jul 2024 14:53:39 +0530
Subject: [PATCH v3 5/5] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 70 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 58 +++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 13 +
.../regress/expected/conflict_resolver.out | 245 ++++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 161 ++++++++-
14 files changed, 929 insertions(+), 24 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 7b536ac6fd..bd0d48ec78 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dbfe0d6b1c..2d75297340 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -1245,6 +1251,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
CloneForeignKeyConstraints(NULL, parent, rel);
+ /* Inherit conflict resolvers configuration from parent. */
+ InheritTableConflictResolvers(rel, parent);
+
table_close(parent, NoLock);
}
@@ -4543,6 +4552,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5116,6 +5127,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5528,6 +5548,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6528,6 +6556,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15650,6 +15682,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16238,6 +16277,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20766,3 +20809,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 42726fe3a6..628b9d10e2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index cbd3a65a27..2384bc53f9 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -279,9 +284,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -484,10 +489,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -515,7 +519,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -604,3 +608,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..636398baf1
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,58 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation
+ * having resolver */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 82f4f5ec49..bbbab5bbc2 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -93,4 +93,17 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
TimestampTz committs);
extern bool CanCreateFullTuple(Relation localrel,
LogicalRepTupleData *newtup);
+
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index d0157ba830..8f0948306b 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,5 +1,8 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_missing | skip
@@ -8,9 +11,7 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(4 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
@@ -19,15 +20,13 @@ SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_missing | error
@@ -39,7 +38,7 @@ select * from pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_missing | skip
@@ -48,3 +47,235 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(4 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable_2;
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1_1 | delete_missing | error | t
+ ptntable_1_1 | insert_exists | remote_apply | t
+ ptntable_1_10 | delete_missing | error | t
+ ptntable_1_10 | insert_exists | remote_apply | t
+ ptntable_1_20 | delete_missing | error | t
+ ptntable_1_20 | insert_exists | remote_apply | t
+(8 rows)
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(8 rows)
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index e3540d7ac8..346590303f 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,27 +1,174 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
v3-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v3-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 4b66fe543e1cc68102313f891914d023a5748ac7 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v3 1/5] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that iolates a NOT DEFERRABLE unique constraint.
update_differ: updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ conflict can only be detected when track_commit_timestamp is
enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 55 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 187 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 50 ++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 44 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 48 ++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 26 +++
src/tools/pgindent/typedefs.list | 1 +
25 files changed, 695 insertions(+), 151 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..ce37fa6490 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,61 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..2a2d188b19 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ingored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..f24f048054
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,187 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3b285894db..06ee46886b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2821,13 +2836,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -3005,13 +3017,13 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5426f1177c..e0ecadd081 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4754,6 +4754,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4826,11 +4827,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4869,6 +4876,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4915,6 +4923,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5155,6 +5165,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..0bc9db991e
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,44 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..c64f17211d 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,15 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..03dabfeb72 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -26,7 +26,12 @@ my $stderr;
# node_A
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
@@ -89,11 +94,32 @@ is( $result, qq(11
'Inserted successfully without leading to infinite recursion in bidirectional replication setup'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update a row that
+# was previously modified by a different source.
+###############################################################################
+
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = true);");
+
+$node_B->safe_psql('postgres', "UPDATE tab SET a = 10 WHERE a = 11;");
+
+$node_A->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
$node_A->safe_psql('postgres', "DELETE FROM tab;");
$node_A->wait_for_catchup($subname_BA);
$node_B->wait_for_catchup($subname_AB);
+# The remaining tests no longer test conflict detection.
+$node_A->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_AB SET (detect_conflict = false);");
+
+$node_A->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_A->restart;
+
###############################################################################
# Check that remote data of node_B (that originated from node_C) is not
# published to node_A.
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e710fa48e5..2485a80833 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -466,6 +466,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v3-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchapplication/octet-stream; name=v3-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchDownload
From 60aeafbaa835499a6049c9c62d578c0d31c17f84 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 25 Jun 2024 15:45:37 +0530
Subject: [PATCH v3 3/5] Implement conflict resolution for INSERT, UPDATE, and
DELETE operations.
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++++-
src/backend/replication/logical/conflict.c | 190 ++++++++++--
src/backend/replication/logical/worker.c | 333 ++++++++++++++++-----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/t/029_on_error.pl | 7 +
6 files changed, 525 insertions(+), 109 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 87b1ea0ab3..a622a0ea8b 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -82,8 +86,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -122,13 +127,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -136,7 +141,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -172,12 +177,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -191,24 +197,29 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -322,6 +333,71 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ *
+ * XXX: Currently, it only handles the simple case of identical table
+ * structures on both Publisher and subscriber. Need to analyze if more
+ * cases can be supported.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -352,7 +428,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
memset(replaces, false, sizeof(replaces));
values[Anum_pg_conflict_conftype - 1] =
- CStringGetTextDatum(stmt->conflict_type);
+ CStringGetTextDatum(stmt->conflict_type);
if (stmt->isReset)
{
@@ -396,3 +472,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 06ee46886b..c4dc03e964 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,33 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+ }
+
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2708,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2834,12 +2906,25 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2972,19 +3057,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3006,6 +3093,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3015,26 +3105,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
+
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
+
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3044,27 +3193,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3105,10 +3286,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3124,19 +3311,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 7eff93eff3..aa21886ee7 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -72,9 +74,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
--
2.34.1
v3-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v3-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From fbf56a37afd3e92d4fa4da30d49cd3f8d13635de Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Mon, 24 Jun 2024 09:55:15 +0530
Subject: [PATCH v3 2/5] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 215 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 20 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 +++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 50 ++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 27 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 467 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4c66..42726fe3a6 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index f24f048054..87b1ea0ab3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -27,6 +36,51 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -34,6 +88,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -185,3 +240,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..a3e5e10d97
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,20 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0bc9db991e..7eff93eff3 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -33,6 +34,41 @@ typedef enum
CT_DELETE_MISSING,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -40,5 +76,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..15c8c0cdba
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,50 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(4 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(4 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 2429ec2bba..76ea2d359b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..e3540d7ac8
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,27 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2485a80833..784194903a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -466,6 +466,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v3-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v3-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 75077a9567540dcf973b9f4dd7840e9bbf28dca3 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Tue, 2 Jul 2024 11:06:28 +0530
Subject: [PATCH v3 4/5] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
This patch also implements last_update_wins resolver.
---
src/backend/replication/logical/conflict.c | 88 ++++++++++++--
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 108 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 26 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 +-
src/include/catalog/pg_conflict.dat | 4 +-
src/include/replication/conflict.h | 3 +
src/include/replication/logicalworker.h | 17 +++
src/include/replication/origin.h | 1 +
src/include/utils/timestamp.h | 1 +
.../regress/expected/conflict_resolver.out | 22 ++--
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 253 insertions(+), 26 deletions(-)
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index a622a0ea8b..cbd3a65a27 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -43,6 +43,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -68,8 +69,8 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR}
};
@@ -79,8 +80,8 @@ const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
* If this changes, change it in pg_conflict.dat as well.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP
};
@@ -211,15 +212,36 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
get_rel_name(conflictidx));
}
else
- return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
+ {
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Key already exists. Applying resolution method \"%s\". The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ ConflictResolverNames[resolver],
+ localorigin, timestamptz_to_str(localts),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Key already exists. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
+ }
+
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
- localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\". The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver],
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -330,6 +352,15 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires %s to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter %s is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -371,6 +402,42 @@ get_conflict_resolver_internal(ConflictType type)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -493,6 +560,9 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c4dc03e964..55fb4f18f5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,19 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = 0;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -987,6 +1000,86 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote commit timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew()
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (replorigin_session_origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, replorigin_session_origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("logical replication clock skew exceeded max tolerated value of %d seconds",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(replorigin_session_origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ elog(LOG, "delaying apply for %ld milliseconds to bring clock skew "
+ "within permissible value of %d seconds",
+ msecs, max_logical_rep_clock_skew);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /* This might change max_logical_rep_clock_skew. */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1008,6 +1101,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Capture the commit timestamp of the remote transaction */
+ replorigin_session_origin_timestamp = begin_data.committime;
+
+ /* Check if there is any clock skew and take configured action */
+ manage_clock_skew();
}
/*
@@ -4669,6 +4768,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4708,10 +4808,12 @@ run_apply_worker()
errmsg("could not connect to the publisher: %s", err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37beeaae..a51f82169e 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d28b0bcb40..6dc23f28cf 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -67,6 +67,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -492,6 +493,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3649,6 +3651,19 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4915,6 +4930,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97e92..3a7fd70506 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,11 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
index a3e5e10d97..2f7fd2394a 100644
--- a/src/include/catalog/pg_conflict.dat
+++ b/src/include/catalog/pg_conflict.dat
@@ -12,8 +12,8 @@
[
-{ conftype => 'insert_exists', confres => 'remote_apply' },
-{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'insert_exists', confres => 'last_update_wins' },
+{ conftype => 'update_differ', confres => 'last_update_wins' },
{ conftype => 'update_missing', confres => 'apply_or_skip' },
{ conftype => 'delete_missing', confres => 'skip' }
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index aa21886ee7..82f4f5ec49 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -54,6 +54,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..2b922f9c62 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,24 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 15c8c0cdba..d0157ba830 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,23 +1,23 @@
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+---------------
+ conftype | confres
+----------------+------------------
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
update_missing | apply_or_skip
(4 rows)
--
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
--
-SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
-SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
ERROR: bbbbb is not a valid conflict resolver
-SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
-RESET CONFLICT RESOLVER for 'ct'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
--
-- Test of SET/RESET CONFLICT RESOLVER with valid names
@@ -40,10 +40,10 @@ RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+----------------
+ conftype | confres
+----------------+------------------
delete_missing | skip
- insert_exists | remote_apply
+ insert_exists | last_update_wins
update_differ | keep_local
update_missing | apply_or_error
(4 rows)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 784194903a..15e5a494ba 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1565,6 +1565,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
On Mon, Jul 1, 2024 at 6:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Mon, Jul 1, 2024 at 1:35 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
Setting resolvers at table-level and subscription-level sounds good to
me. DDLs for setting resolvers at subscription-level would need the
subscription name to be specified?Yes, it should be part of the ALTER/CREATE SUBSCRIPTION command. One
idea could be to have syntax as follows:ALTER SUBSCRIPTION name SET CONFLICT RESOLVER 'conflict_resolver' FOR
'conflict_type';
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR 'conflict_type';CREATE SUBSCRIPTION subscription_name CONNECTION 'conninfo'
PUBLICATION publication_name [, ...] CONFLICT RESOLVER
'conflict_resolver' FOR 'conflict_type';
Looks good to me.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
On Wed, Jul 3, 2024 at 5:06 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:48 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Jul 3, 2024 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
What I am saying is that if we are not applying the whole transaction,
it means we are not receiving it either unless we plan to spill it to
a file. If we don't spill it to a file, the network buffer will fill
up very quickly. This issue wouldn't occur if we waited right before
the commit because, by that time, we would have already received all
the data from the network.We would have received the transaction data but there could be other
transactions that need to wait because the apply worker is waiting
before the commit.Yeah, that's a valid point, can parallel apply worker help here?
So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.Yes, spilling to file or cap on the wait time should help, and as I
said above maybe a parallel apply worker can also help.
It is not clear to me how a parallel apply worker can help in this
case. Can you elaborate on what you have in mind?
--
With Regards,
Amit Kapila.
On Thu, Jul 4, 2024 at 5:37 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.Yes, spilling to file or cap on the wait time should help, and as I
said above maybe a parallel apply worker can also help.It is not clear to me how a parallel apply worker can help in this
case. Can you elaborate on what you have in mind?
If we decide to wait at commit time, and before starting to apply if
we already see a remote commit_ts clock is ahead, then if we apply
such transactions using the parallel worker, wouldn't it solve the
issue of the network buffer congestion? Now the apply worker can move
ahead and fetch new transactions from the buffer as our waiting
transaction will not block it. I understand that if this transaction
is going to wait at commit then any future transaction that we are
going to fetch might also going to wait again because if the previous
transaction committed before is in the future then the subsequent
transaction committed after this must also be in future so eventually
that will also go to some another parallel worker and soon we end up
consuming all the parallel worker if the clock skew is large. So I
won't say this will resolve the problem and we would still have to
fall back to the spilling to the disk but that's just in the worst
case when the clock skew is really huge. In most cases which is due
to slight clock drift by the time we apply the medium to large size
transaction, the local clock should be able to catch up the remote
commit_ts and we might not have to wait in most of the cases.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Please find the new patch set (v4). It implements the resolvers for
conflict type : 'delete_differ'.
Supported resolutions for ‘delete_differ’ are :
- ‘last_update_wins': Apply the change with the latest timestamp (default)
- 'remote_apply': Apply the remote delete.
- 'keep_local': Skip the remote delete and continue.
- 'error': The apply worker will error out and restart.
The changes made in the patches are as follows:
- Updated the conflict detection patch (patch0001) to the latest
version from [1]/messages/by-id/OS0PR01MB571686E464A325F26CEFCCEF94DD2@OS0PR01MB5716.jpnprd01.prod.outlook.com, which implements delete_differ conflict detection.
- Patch0002 now supports resolver settings for delete_differ.
- Patch0003 implements resolutions for delete_differ as well.
- Patch0004 includes changes to support last_update_wins resolution
for delete_differ.
[1]: /messages/by-id/OS0PR01MB571686E464A325F26CEFCCEF94DD2@OS0PR01MB5716.jpnprd01.prod.outlook.com
--
Thanks,
Nisha
Attachments:
v4-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v4-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 3c5017cd591ee1f289607ec936bfb8798615074d Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v4 1/5] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 23 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 66 +++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 191 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 79 ++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 47 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 ++++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 801 insertions(+), 152 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ccdd24312b..f078d6364b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1565,7 +1565,9 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped, but note that this scenario can be reported in the server log
+ if <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled.
</para>
<para>
@@ -1634,6 +1636,25 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row. Refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for other conflicts that will be logged when enabling <literal>detect_conflict</literal>.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..caa523b9bd 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,72 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..b90e64b05b
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,191 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3b285894db..b4c834ed3f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2810,6 +2825,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2821,13 +2850,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2994,6 +3020,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3005,16 +3034,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5426f1177c..e0ecadd081 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4754,6 +4754,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4826,11 +4827,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4869,6 +4876,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4915,6 +4923,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5155,6 +5165,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..a9f521aaca
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..8c929c07c7 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e710fa48e5..2485a80833 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -466,6 +466,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v4-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v4-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From 07df9b3843d9b8d72b1448f2ed83185fc46804fc Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 10:23:58 +0530
Subject: [PATCH v4 2/5] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
- delete_differ
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 218 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 21 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 ++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 54 +++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 28 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 476 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4c66..42726fe3a6 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b90e64b05b..93d9659752 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -28,6 +37,54 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_DIFFER] = "delete_differ"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -35,6 +92,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -189,3 +247,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..8d6183e3f2
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,21 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' },
+{ conftype => 'delete_differ', confres => 'remote_apply' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index a9f521aaca..c6023e0914 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -36,6 +37,41 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_DIFFER
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -43,5 +79,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..53ecb430fa
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,54 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(5 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 2429ec2bba..76ea2d359b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..f83d14b229
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,28 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2485a80833..784194903a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -466,6 +466,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v4-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchapplication/octet-stream; name=v4-0003-Implement-conflict-resolution-for-INSERT-UPDATE-a.patchDownload
From 816d6bf0e2dee0394190511ad9c091e39c97302e Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 15:46:58 +0530
Subject: [PATCH v4 3/5] Implement conflict resolution for INSERT, UPDATE, and
DELETE operations.
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: remote_apply, keep_local, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++++-
src/backend/replication/logical/conflict.c | 195 +++++++++--
src/backend/replication/logical/worker.c | 361 ++++++++++++++++-----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/t/029_on_error.pl | 7 +
6 files changed, 541 insertions(+), 126 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 93d9659752..d215d48dc6 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -86,8 +90,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -126,13 +131,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -140,7 +145,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -176,12 +181,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -195,27 +201,35 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -329,6 +343,71 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ *
+ * XXX: Currently, it only handles the simple case of identical table
+ * structures on both Publisher and subscriber. Need to analyze if more
+ * cases can be supported.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -403,3 +482,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b4c834ed3f..239d5c7cd1 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,35 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2685,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2710,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2815,6 +2889,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2836,24 +2912,45 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, resolver, localrel, InvalidOid,
localxmin, localorigin, localts, NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
+
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2986,19 +3083,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3023,6 +3122,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3032,38 +3134,81 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /*
- * If conflict detection is enabled, check whether the local
- * tuple was modified by a different origin. If detected,
- * report the conflict.
- */
- if (MySubscription->detectconflict &&
- GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
- InvalidOid, localxmin, localorigin,
- localts, NULL);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3073,27 +3218,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3134,10 +3311,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3153,19 +3336,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c6023e0914..dcc89f222f 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -75,9 +77,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
--
2.34.1
v4-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v4-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 62b5dd1c41d30888a37b5365ad35f2603eb2b704 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 4 Jul 2024 16:16:46 +0530
Subject: [PATCH v4 4/5] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
This patch also implements last_update_wins resolver.
---
src/backend/replication/logical/conflict.c | 99 ++++++++++++++--
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 108 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 26 +++++
src/backend/utils/misc/postgresql.conf.sample | 6 +-
src/include/catalog/pg_conflict.dat | 6 +-
src/include/replication/conflict.h | 3 +
src/include/replication/logicalworker.h | 17 +++
src/include/replication/origin.h | 1 +
src/include/utils/timestamp.h | 1 +
.../regress/expected/conflict_resolver.out | 16 +--
src/tools/pgindent/typedefs.list | 1 +
13 files changed, 259 insertions(+), 27 deletions(-)
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index d215d48dc6..4934099cc8 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -44,6 +44,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -69,11 +70,11 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -81,11 +82,11 @@ const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
* If this changes, change it in pg_conflict.dat as well.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+ [CT_DELETE_DIFFER] = CR_LAST_UPDATE_WINS
};
@@ -215,11 +216,30 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
get_rel_name(conflictidx));
}
else
- return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
+ {
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Key already exists. Applying resolution method \"%s\". The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ ConflictResolverNames[resolver],
+ localorigin, timestamptz_to_str(localts),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Key already exists. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
+ }
+
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
- localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\". The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver],
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"",
ConflictResolverNames[resolver]);
@@ -227,9 +247,16 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"",
ConflictResolverNames[resolver]);
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
- localorigin, localxmin, timestamptz_to_str(localts),
- ConflictResolverNames[resolver]);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\". The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver],
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -340,6 +367,15 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires %s to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter %s is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -381,6 +417,42 @@ get_conflict_resolver_internal(ConflictType type)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -503,6 +575,9 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 239d5c7cd1..7586aa53ae 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,19 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = 0;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -987,6 +1000,86 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote commit timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew()
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (replorigin_session_origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, replorigin_session_origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("logical replication clock skew exceeded max tolerated value of %d seconds",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(replorigin_session_origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ elog(LOG, "delaying apply for %ld milliseconds to bring clock skew "
+ "within permissible value of %d seconds",
+ msecs, max_logical_rep_clock_skew);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /* This might change max_logical_rep_clock_skew. */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1008,6 +1101,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Capture the commit timestamp of the remote transaction */
+ replorigin_session_origin_timestamp = begin_data.committime;
+
+ /* Check if there is any clock skew and take configured action */
+ manage_clock_skew();
}
/*
@@ -4694,6 +4793,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4733,10 +4833,12 @@ run_apply_worker()
errmsg("could not connect to the publisher: %s", err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37beeaae..a51f82169e 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index d28b0bcb40..6dc23f28cf 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -67,6 +67,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -492,6 +493,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3649,6 +3651,19 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4915,6 +4930,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97e92..3a7fd70506 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,11 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
index 8d6183e3f2..1905151840 100644
--- a/src/include/catalog/pg_conflict.dat
+++ b/src/include/catalog/pg_conflict.dat
@@ -12,10 +12,10 @@
[
-{ conftype => 'insert_exists', confres => 'remote_apply' },
-{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'insert_exists', confres => 'last_update_wins' },
+{ conftype => 'update_differ', confres => 'last_update_wins' },
{ conftype => 'update_missing', confres => 'apply_or_skip' },
{ conftype => 'delete_missing', confres => 'skip' },
-{ conftype => 'delete_differ', confres => 'remote_apply' }
+{ conftype => 'delete_differ', confres => 'last_update_wins' }
]
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index dcc89f222f..d38a22ddde 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -57,6 +57,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..2b922f9c62 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,24 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 53ecb430fa..c21486dbb4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,11 +1,11 @@
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+---------------
- delete_differ | remote_apply
+ conftype | confres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
update_missing | apply_or_skip
(5 rows)
@@ -43,11 +43,11 @@ RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+----------------
+ conftype | confres
+----------------+------------------
delete_differ | keep_local
delete_missing | skip
- insert_exists | remote_apply
+ insert_exists | last_update_wins
update_differ | keep_local
update_missing | apply_or_error
(5 rows)
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 784194903a..15e5a494ba 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1565,6 +1565,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
v4-0005-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v4-0005-Configure-table-level-conflict-resolvers.patchDownload
From 2f7f588de7b0a7c028586bc59075e5cdccd3c521 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Fri, 5 Jul 2024 11:29:45 +0530
Subject: [PATCH v4 5/5] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 70 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 58 +++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 13 +
.../regress/expected/conflict_resolver.out | 245 ++++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 161 ++++++++-
14 files changed, 929 insertions(+), 24 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 2983b9180f..1ba0f74a2d 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dbfe0d6b1c..2d75297340 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -1245,6 +1251,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
CloneForeignKeyConstraints(NULL, parent, rel);
+ /* Inherit conflict resolvers configuration from parent. */
+ InheritTableConflictResolvers(rel, parent);
+
table_close(parent, NoLock);
}
@@ -4543,6 +4552,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5116,6 +5127,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5528,6 +5548,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6528,6 +6556,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15650,6 +15682,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16238,6 +16277,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20766,3 +20809,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 42726fe3a6..628b9d10e2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 4934099cc8..0b4f0e25a9 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -294,9 +299,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -499,10 +504,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -530,7 +534,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -619,3 +623,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..636398baf1
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,58 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation
+ * having resolver */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index d38a22ddde..3ba83a25cd 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -96,4 +96,17 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
TimestampTz committs);
extern bool CanCreateFullTuple(Relation localrel,
LogicalRepTupleData *newtup);
+
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index c21486dbb4..802a7583d4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,5 +1,8 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | last_update_wins
@@ -9,9 +12,7 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(5 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
@@ -20,16 +21,14 @@ SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_differ | keep_local
@@ -42,7 +41,7 @@ select * from pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | keep_local
@@ -52,3 +51,235 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(5 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable_2;
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1_1 | delete_missing | error | t
+ ptntable_1_1 | insert_exists | remote_apply | t
+ ptntable_1_10 | delete_missing | error | t
+ ptntable_1_10 | insert_exists | remote_apply | t
+ ptntable_1_20 | delete_missing | error | t
+ ptntable_1_20 | insert_exists | remote_apply | t
+(8 rows)
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(8 rows)
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index f83d14b229..b4574d1df4 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,17 +1,19 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
@@ -19,10 +21,155 @@ SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
On Fri, Jul 5, 2024 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Thu, Jul 4, 2024 at 5:37 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.Yes, spilling to file or cap on the wait time should help, and as I
said above maybe a parallel apply worker can also help.It is not clear to me how a parallel apply worker can help in this
case. Can you elaborate on what you have in mind?If we decide to wait at commit time, and before starting to apply if
we already see a remote commit_ts clock is ahead, then if we apply
such transactions using the parallel worker, wouldn't it solve the
issue of the network buffer congestion? Now the apply worker can move
ahead and fetch new transactions from the buffer as our waiting
transaction will not block it. I understand that if this transaction
is going to wait at commit then any future transaction that we are
going to fetch might also going to wait again because if the previous
transaction committed before is in the future then the subsequent
transaction committed after this must also be in future so eventually
that will also go to some another parallel worker and soon we end up
consuming all the parallel worker if the clock skew is large. So I
won't say this will resolve the problem and we would still have to
fall back to the spilling to the disk but that's just in the worst
case when the clock skew is really huge. In most cases which is due
to slight clock drift by the time we apply the medium to large size
transaction, the local clock should be able to catch up the remote
commit_ts and we might not have to wait in most of the cases.
Yeah, this is possible but even if go with the spilling logic at first
it should work for all cases. If we get some complaints then we can
explore executing such transactions by parallel apply workers.
Personally, I am of the opinion that clock synchronization should be
handled outside the database system via network time protocols like
NTP. Still, we can have some simple solution to inform users about the
clock_skew.
--
With Regards,
Amit Kapila.
On Fri, Jul 5, 2024 at 2:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Jul 5, 2024 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Thu, Jul 4, 2024 at 5:37 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
So, the situation will be the same. We can even
decide to spill the data to files if the decision is that we need to
wait to avoid network buffer-fill situations. But note that the wait
in apply worker has consequences that the subscriber won't be able to
confirm the flush position and publisher won't be able to vacuum the
dead rows and we won't be remove WAL as well. Last time when we
discussed the delay_apply feature, we decided not to proceed because
of such issues. This is the reason I proposed a cap on wait time.Yes, spilling to file or cap on the wait time should help, and as I
said above maybe a parallel apply worker can also help.It is not clear to me how a parallel apply worker can help in this
case. Can you elaborate on what you have in mind?If we decide to wait at commit time, and before starting to apply if
we already see a remote commit_ts clock is ahead, then if we apply
such transactions using the parallel worker, wouldn't it solve the
issue of the network buffer congestion? Now the apply worker can move
ahead and fetch new transactions from the buffer as our waiting
transaction will not block it. I understand that if this transaction
is going to wait at commit then any future transaction that we are
going to fetch might also going to wait again because if the previous
transaction committed before is in the future then the subsequent
transaction committed after this must also be in future so eventually
that will also go to some another parallel worker and soon we end up
consuming all the parallel worker if the clock skew is large. So I
won't say this will resolve the problem and we would still have to
fall back to the spilling to the disk but that's just in the worst
case when the clock skew is really huge. In most cases which is due
to slight clock drift by the time we apply the medium to large size
transaction, the local clock should be able to catch up the remote
commit_ts and we might not have to wait in most of the cases.Yeah, this is possible but even if go with the spilling logic at first
it should work for all cases. If we get some complaints then we can
explore executing such transactions by parallel apply workers.
Personally, I am of the opinion that clock synchronization should be
handled outside the database system via network time protocols like
NTP. Still, we can have some simple solution to inform users about the
clock_skew.
Yeah, that makes sense, in the first version we can have a simple
solution and we can further improvise it based on the feedback.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Mon, Jul 1, 2024 at 1:17 PM Ajin Cherian <itsajin@gmail.com> wrote:
On Thu, Jun 27, 2024 at 1:14 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Please find the attached 'patch0003', which implements conflict
resolutions according to the global resolver settings.Summary of Conflict Resolutions Implemented in 'patch0003':
INSERT Conflicts:
------------------------
1) Conflict Type: 'insert_exists'Supported Resolutions:
a) 'remote_apply': Convert the INSERT to an UPDATE and apply.
b) 'keep_local': Ignore the incoming (conflicting) INSERT and retain
the local tuple.
c) 'error': The apply worker will error out and restart.Hi Nisha,
While testing the patch, when conflict resolution is configured and insert_exists is set to "remote_apply", I see this warning in the logs due to a resource not being closed:
2024-07-01 02:52:59.427 EDT [20304] LOG: conflict insert_exists detected on relation "public.test1"
2024-07-01 02:52:59.427 EDT [20304] DETAIL: Key already exists. Applying resolution method "remote_apply"
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for replication origin "pg_16417" during message type "INSERT" for replication target relation "public.test1" in transaction 763, finished at 0/15E7F68
2024-07-01 02:52:59.427 EDT [20304] WARNING: resource was not closed: [138] (rel=base/5/16413, blockNum=0, flags=0x93800000, refcount=1 1)
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for replication origin "pg_16417" during message type "COMMIT" in transaction 763, finished at 0/15E7F68
2024-07-01 02:52:59.427 EDT [20304] WARNING: resource was not closed: TupleDesc 0x7f8c0439e448 (16402,-1)
2024-07-01 02:52:59.427 EDT [20304] CONTEXT: processing remote data for replication origin "pg_16417" during message type "COMMIT" in transaction 763, finished at 0/15E7F68
Thank you Ajin for reporting the issue, This is now fixed with the
v4-0003 patch.
--
Thanks,
Nisha
Hi,
I researched about how to detect the resolve update_deleted and thought
about one idea: which is to maintain the xmin in logical slot to preserve
the dead row and support latest_timestamp_xmin resolution for
update_deleted to maintain data consistency.
Here are details of the xmin idea and resolution of update_deleted:
1. how to preserve the dead row so that we can detect update_delete
conflict correctly. (In the following explanation, let's assume there is a
a multimeter setup with node A, B).
To preserve the dead row on node A, I think we could maintain the "xmin"
in the logical replication slot on Node A to prevent the VACCUM from
removing the dead row in user table. The walsender that acquires the slot
is responsible to advance the xmin. (Node that I am trying to explore
xmin idea as it could be more efficient than using commit_timestamp, and the
logic could be simpler as we are already maintaining catalog_xmin in
logical slot and xmin in physical slot)
- Strategy for advancing xmin:
The xmin can be advanced if a) a transaction (xid:1000) has been flushed
to the remote node (Node B in this case). *AND* b) On Node B, the local
transactions that happened before applying the remote
transaction(xid:1000) were also sent and flushed to the Node A.
- The implementation:
condition a) can be achieved with existing codes, the walsender can
advance the xmin similar to the catalog_xmin.
For condition b), we can add a subscription option (say 'feedback_slot').
The feedback_slot indicates the replication slot that will send changes to
the origin (On Node B, the slot should be subBA). The apply worker will
check the status(confirmed flush lsn) of the 'feedback slot' and send
feedback to the walsender about the WAL position that has been sent and
flushed via the feedback_slot.
For example, on Node B, we specify the replication slot (subBA) that is
sending changes to Node A. The apply worker on Node B will send
feedback(WAL position that has been sent to the Node A) to Node A
regularly. Then the Node A can use the position to advance the xmin.
(Similar to the hot_standby_feedback).
2. The resolution for update_delete
The current design doesn't support 'last_timestamp_win'. But this could be
a problem if update_deleted is detected due to some very old dead row.
Assume the update has the latest timestamp, and if we skip the update due
to these very old dead rows, the data would be inconsistent because the
latest update data is missing.
The ideal resolution should compare the timestamp of the UPDATE and the
timestamp of the transaction that produced these dead rows. If the UPDATE
is newer, the convert the UDPATE to INSERT, otherwise, skip the UPDATE.
Best Regards,
Hou zj
On Monday, July 8, 2024 12:32 PM Zhijie Hou (Fujitsu) <houzj.fnst@fujitsu.com> wrote:
I researched about how to detect the resolve update_deleted and thought
about one idea: which is to maintain the xmin in logical slot to preserve
the dead row and support latest_timestamp_xmin resolution for
update_deleted to maintain data consistency.Here are details of the xmin idea and resolution of update_deleted:
1. how to preserve the dead row so that we can detect update_delete
conflict correctly. (In the following explanation, let's assume there is a
a multimeter setup with node A, B).To preserve the dead row on node A, I think we could maintain the "xmin"
in the logical replication slot on Node A to prevent the VACCUM from
removing the dead row in user table. The walsender that acquires the slot
is responsible to advance the xmin. (Node that I am trying to explore
xmin idea as it could be more efficient than using commit_timestamp, and the
logic could be simpler as we are already maintaining catalog_xmin in
logical slot and xmin in physical slot)- Strategy for advancing xmin:
The xmin can be advanced if a) a transaction (xid:1000) has been flushed
to the remote node (Node B in this case). *AND* b) On Node B, the local
transactions that happened before applying the remote
transaction(xid:1000) were also sent and flushed to the Node A.- The implementation:
condition a) can be achieved with existing codes, the walsender can
advance the xmin similar to the catalog_xmin.For condition b), we can add a subscription option (say 'feedback_slot').
The feedback_slot indicates the replication slot that will send changes to
the origin (On Node B, the slot should be subBA). The apply worker will
check the status(confirmed flush lsn) of the 'feedback slot' and send
feedback to the walsender about the WAL position that has been sent and
flushed via the feedback_slot.
The above are some initial thoughts of how to preserve the dead row for
update_deleted conflict detection.
After thinking more, I have identified a few additional cases that I
missed to analyze regarding the design. One aspect that needs more
thoughts is the possibility of multiple slots on each node. In this
scenario, the 'feedback_slot' subscription option would need to be
structured as a list. However, requiring users to specify all the slots
may not be user-friendly. I will explore if this process can be
automated.
In addition, I will think more about the potential impact of re-using the
existing 'xmin' of the slot which may affect existing logic that relies on
'xmin'.
I will analyze more and reply about these points.
Best Regards,
Hou zj
On Fri, Jul 5, 2024 at 5:12 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Thank you Ajin for reporting the issue, This is now fixed with the
v4-0003 patch.
Please find v5 patch-set. Changes are:
1) patch003:
Added test cases for all resolvers (034_conflict_resolver.pl).
2) Patch004:
a) Emit error while resolving conflict if conflict resolver is default
'last_update_wins' but track_commit_timetsamp is not enabled.
b) Emit Warning during create and alter subscription when
'detect_conflict' is ON but 'track_commit_timetsamp' is not enabled.
c) Restrict start of pa worker if either max-clock-skew is configured
or conflict detection and resolution is enabled for a subscription.
d) Implement clock-skew delay/error when changes are applied from a
file (apply_spooled_messages).
e) Implement clock-skew delay while applying prepared changes (two
phase txns). The prepare-timestamp to be considered as base for
clock-skew handling as well as for last_update_win resolver.
<TODO: This needs to be analyzed and tested further to see if there is
any side effect of taking prepare-timestamp as base.>
Thanks Ajin fo working on 1.
Thanks Nisha for working on 2a,2b.
thanks
Shveta
Attachments:
v5-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v5-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From 19c532794028e833ed2b210f5ddf6379cd5574e7 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 10:23:58 +0530
Subject: [PATCH v5 2/5] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
- delete_differ
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 218 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 21 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 ++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 54 +++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 28 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 476 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4c66..42726fe3a6 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b90e64b05b..93d9659752 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -28,6 +37,54 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_DIFFER] = "delete_differ"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -35,6 +92,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -189,3 +247,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..8d6183e3f2
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,21 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' },
+{ conftype => 'delete_differ', confres => 'remote_apply' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index a9f521aaca..c6023e0914 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -36,6 +37,41 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_DIFFER
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -43,5 +79,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..53ecb430fa
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,54 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(5 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 2429ec2bba..76ea2d359b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..f83d14b229
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,28 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3c79446ebb..c920a59525 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v5-0003-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v5-0003-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 51ca8c0c0568510807270f41d2ebde95943bf384 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 15:46:58 +0530
Subject: [PATCH v5 3/5] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: remote_apply, keep_local, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 80 ++-
src/backend/replication/logical/conflict.c | 191 ++++++-
src/backend/replication/logical/worker.c | 361 +++++++++----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 19 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/029_on_error.pl | 7 +
.../subscription/t/034_conflict_resolver.pl | 475 ++++++++++++++++++
8 files changed, 1013 insertions(+), 126 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..3e8d768214 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,82 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(*conflictslot, rel,
+ CT_INSERT_EXISTS,
+ &apply_remote, NULL, uniqueidx,
+ xmin, origin, committs);
+
+ ReportApplyConflict(LOG, CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +693,15 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, 0, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 93d9659752..5a7f0e0b53 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -86,8 +90,9 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
TupleTableSlot *conflictslot);
@@ -126,13 +131,13 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(int elevel, ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
{
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
@@ -140,7 +145,7 @@ ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
localts, conflictslot));
}
@@ -176,12 +181,13 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
{
switch (type)
{
@@ -195,27 +201,35 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (!resolver || (resolver == CR_ERROR))
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -329,6 +343,67 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -403,3 +478,65 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ * It emits error in case the resolver is set to 'ERROR'.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin, TimestampTz committs)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ *apply_remote = false;
+ elog(LOG, "UPDATE can not be converted to INSERT, hence SKIP the update!");
+ break;
+ }
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ {
+ *apply_remote = true;
+ break;
+ }
+ else
+ {
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ }
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ *apply_remote = false;
+ break;
+ case CR_ERROR:
+ ReportApplyConflict(ERROR, type, resolver, localrel, conflictidx,
+ xmin, origin, committs, localslot);
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method", ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b4c834ed3f..239d5c7cd1 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,35 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2685,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2710,73 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ {
+ elog(LOG, "Converting UPDATE to INSERT and applying!");
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
+
+ }
}
/* Cleanup. */
@@ -2815,6 +2889,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2836,24 +2912,45 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, resolver, localrel, InvalidOid,
localxmin, localorigin, localts, NULL);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
+
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+ }
}
/* Cleanup. */
@@ -2986,19 +3083,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3023,6 +3122,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3032,38 +3134,81 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0);
+
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
InvalidRepOriginId, 0, NULL);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL, InvalidOid,
+ localxmin, localorigin, localts);
- /*
- * If conflict detection is enabled, check whether the local
- * tuple was modified by a different origin. If detected,
- * report the conflict.
- */
- if (MySubscription->detectconflict &&
- GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
- InvalidOid, localxmin, localorigin,
- localts, NULL);
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL);
+ }
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3073,27 +3218,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3134,10 +3311,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3153,19 +3336,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c6023e0914..dcc89f222f 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -75,9 +77,20 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
- Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ ConflictResolver resolver, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid conflictidx, TransactionId xmin,
+ RepOriginId origin,
+ TimestampTz committs);
+extern bool CanCreateFullTuple(Relation localrel,
+ LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..ae8030aac6
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,475 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on);");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT conftype, confres FROM pg_conflict ORDER BY conftype"
+);
+is( $result, qq(delete_differ|remote_apply
+delete_missing|skip
+insert_exists|remote_apply
+update_differ|remote_apply
+update_missing|apply_or_skip),
+ "confirm that the default conflict resolvers are in place"
+);
+
+############################################
+# Test 'remote_apply' for 'insert_exists'
+############################################
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);"
+);
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'insert_exists';"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=3);");
+
+# XXX: Fixme. It intermittently fails to capture this log.
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+#$node_subscriber->wait_for_log(
+# qr/LOG: conflict delete_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'delete_missing';"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);"
+);
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'keep_local' for 'update_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'update_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);"
+);
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: UPDATE can not be converted to INSERT, hence SKIP the update!/, $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);"
+);
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'skip' for 'update_missing';"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);"
+);
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'update_missing';"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+done_testing();
--
2.34.1
v5-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v5-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 36acc99dbfe4c6ed9b846dee067839954ce6778d Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 4 Jul 2024 16:16:46 +0530
Subject: [PATCH v5 4/5] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
This patch also implements last_update_wins resolver.
---
src/backend/commands/subscriptioncmds.c | 18 +++
.../replication/logical/applyparallelworker.c | 30 +++-
src/backend/replication/logical/conflict.c | 107 ++++++++++++--
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 130 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 26 ++++
src/backend/utils/misc/postgresql.conf.sample | 6 +-
src/include/catalog/pg_conflict.dat | 6 +-
src/include/replication/conflict.h | 3 +
src/include/replication/logicalworker.h | 17 +++
src/include/replication/origin.h | 1 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
.../regress/expected/conflict_resolver.out | 16 +--
src/test/regress/expected/subscription.out | 2 +
.../subscription/t/034_conflict_resolver.pl | 16 ++-
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 349 insertions(+), 35 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e670d72708..9126255ff3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/xact.h"
@@ -671,6 +672,13 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
#endif
+ /* Warn if detect_conflict is enabled and track_commit_timestamp is off */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
+
rel = table_open(SubscriptionRelationId, RowExclusiveLock);
/* Check if name is used */
@@ -1277,6 +1285,16 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ /*
+ * Warn if detect_conflict is enabled and
+ * track_commit_timestamp is off.
+ */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
}
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..ab4ede3fa9 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -312,6 +312,24 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Don't start a new parallel worker if user has set max clock skew
+ * tolerance as the commit timestamp will be needed during 'begin' itself
+ * to manage clock skew.
+ *
+ * Also don't start parallel apply worker if conflict detection and
+ * resolution is ON as commit timesamp will be needed for time based
+ * resolution methods while applying concerned changes.
+ *
+ * XXX: For second case, see if we can reduce the scope of this
+ * restriction to only such cases where time-based resolvers are actually
+ * being used.
+ */
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ MySubscription->detectconflict)
+ return false;
+
+
return true;
}
@@ -696,9 +714,19 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured, thus it is okay to pass 0 as origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with conflict
+ * detection enabled, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5a7f0e0b53..c330b21217 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -44,6 +44,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -69,11 +70,11 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -81,11 +82,11 @@ const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
* If this changes, change it in pg_conflict.dat as well.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+ [CT_DELETE_DIFFER] = CR_LAST_UPDATE_WINS
};
@@ -215,11 +216,30 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
get_rel_name(conflictidx));
}
else
- return errdetail("Key already exists. Applying resolution method \"%s\"", ConflictResolverNames[resolver]);
+ {
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Key already exists. Applying resolution method \"%s\". The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ ConflictResolverNames[resolver],
+ localorigin, timestamptz_to_str(localts),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Key already exists. Applying resolution method \"%s\"",
+ ConflictResolverNames[resolver]);
+ }
+
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
- localorigin, localxmin, timestamptz_to_str(localts), ConflictResolverNames[resolver]);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\". The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver],
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
case CT_UPDATE_MISSING:
return errdetail("Did not find the row to be updated. Applying resolution method \"%s\"",
ConflictResolverNames[resolver]);
@@ -227,9 +247,16 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
return errdetail("Did not find the row to be deleted. Applying resolution method \"%s\"",
ConflictResolverNames[resolver]);
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
- localorigin, localxmin, timestamptz_to_str(localts),
- ConflictResolverNames[resolver]);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\". The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver],
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
+ else
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s. Applying resolution method \"%s\"",
+ localorigin, localxmin, timestamptz_to_str(localts),
+ ConflictResolverNames[resolver]);
}
return 0; /* silence compiler warning */
@@ -340,6 +367,15 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -381,6 +417,42 @@ get_conflict_resolver_internal(ConflictType type)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -499,6 +571,17 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 239d5c7cd1..3156b0773f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,19 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -987,6 +1000,86 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote commit timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("logical replication clock skew exceeded max tolerated value of %d seconds",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ elog(LOG, "delaying apply for %ld milliseconds to bring clock skew "
+ "within permissible value of %d seconds",
+ msecs, max_logical_rep_clock_skew);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /* This might change max_logical_rep_clock_skew. */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1008,6 +1101,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
+ /* Capture the commit timestamp of the remote transaction */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -1065,6 +1164,12 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
+
+ /* Capture the prepare timestamp of the remote transaction */
+ replorigin_session_origin_timestamp = begin_data.prepare_time;
}
/*
@@ -1305,7 +1410,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2002,7 +2108,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2055,6 +2162,13 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
+ /* Check if there is any clock skew and take configured action */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /* Capture the timestamp (prepare or commit) of the remote transaction */
+ replorigin_session_origin_timestamp = origin_timestamp;
+
/*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
@@ -2160,7 +2274,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -4694,6 +4809,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4733,10 +4849,12 @@ run_apply_worker()
errmsg("could not connect to the publisher: %s", err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37beeaae..a51f82169e 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 630ed0f162..f6911ac4a9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -67,6 +67,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -492,6 +493,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3649,6 +3651,19 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4915,6 +4930,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97e92..3a7fd70506 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,11 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
index 8d6183e3f2..1905151840 100644
--- a/src/include/catalog/pg_conflict.dat
+++ b/src/include/catalog/pg_conflict.dat
@@ -12,10 +12,10 @@
[
-{ conftype => 'insert_exists', confres => 'remote_apply' },
-{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'insert_exists', confres => 'last_update_wins' },
+{ conftype => 'update_differ', confres => 'last_update_wins' },
{ conftype => 'update_missing', confres => 'apply_or_skip' },
{ conftype => 'delete_missing', confres => 'skip' },
-{ conftype => 'delete_differ', confres => 'remote_apply' }
+{ conftype => 'delete_differ', confres => 'last_update_wins' }
]
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index dcc89f222f..d38a22ddde 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -57,6 +57,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..2b922f9c62 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,24 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 515aefd519..dc9e067fac 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -267,7 +267,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 53ecb430fa..c21486dbb4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,11 +1,11 @@
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+---------------
- delete_differ | remote_apply
+ conftype | confres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
update_missing | apply_or_skip
(5 rows)
@@ -43,11 +43,11 @@ RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+----------------
+ conftype | confres
+----------------+------------------
delete_differ | keep_local
delete_missing | skip
- insert_exists | remote_apply
+ insert_exists | last_update_wins
update_differ | keep_local
update_missing | apply_or_error
(5 rows)
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index a8b0086dd9..47c4dccc3e 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -433,6 +433,8 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index ae8030aac6..6f1479dff7 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -51,10 +51,10 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
my $result = $node_subscriber->safe_psql('postgres',
"SELECT conftype, confres FROM pg_conflict ORDER BY conftype"
);
-is( $result, qq(delete_differ|remote_apply
+is( $result, qq(delete_differ|last_update_wins
delete_missing|skip
-insert_exists|remote_apply
-update_differ|remote_apply
+insert_exists|last_update_wins
+update_differ|last_update_wins
update_missing|apply_or_skip),
"confirm that the default conflict resolvers are in place"
);
@@ -63,6 +63,11 @@ update_missing|apply_or_skip),
# Test 'remote_apply' for 'insert_exists'
############################################
+# Change CONFLICT RESOLVER of insert_exists to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'remote_apply' for 'insert_exists';"
+);
+
# Create local data on the subscriber
$node_subscriber->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
@@ -202,6 +207,11 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
# Test 'remote_apply' for 'update_differ'
#########################################
+# Change CONFLICT RESOLVER of update_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'remote_apply' for 'update_differ';"
+);
+
# Insert data in the publisher
$node_publisher->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c920a59525..13cc7a33fb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1566,6 +1566,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
v5-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v5-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 3fd759ad7ce87a573730c27ec61e7d392d47aa4d Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v5 1/5] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 23 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 66 +++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 191 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 79 ++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 47 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 ++++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 801 insertions(+), 152 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ccdd24312b..f078d6364b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1565,7 +1565,9 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped, but note that this scenario can be reported in the server log
+ if <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled.
</para>
<para>
@@ -1634,6 +1636,25 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row. Refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for other conflicts that will be logged when enabling <literal>detect_conflict</literal>.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..caa523b9bd 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,72 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index e407428dbc..e670d72708 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1146,7 +1162,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1256,6 +1272,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..b90e64b05b
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,191 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3b285894db..b4c834ed3f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2810,6 +2825,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2821,13 +2850,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2994,6 +3020,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3005,16 +3034,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..a9f521aaca
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 0f2a25cdc1..a8b0086dd9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..8c929c07c7 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 635e6d6e21..3c79446ebb 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v5-0005-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v5-0005-Configure-table-level-conflict-resolvers.patchDownload
From b7bddbe815109f7fda24d8d1ca3fc2b097a55424 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Tue, 9 Jul 2024 14:37:08 +0530
Subject: [PATCH v5 5/5] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 70 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 58 +++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 13 +
.../regress/expected/conflict_resolver.out | 245 ++++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 161 ++++++++-
14 files changed, 929 insertions(+), 24 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 2983b9180f..1ba0f74a2d 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index dbfe0d6b1c..2d75297340 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -1245,6 +1251,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
CloneForeignKeyConstraints(NULL, parent, rel);
+ /* Inherit conflict resolvers configuration from parent. */
+ InheritTableConflictResolvers(rel, parent);
+
table_close(parent, NoLock);
}
@@ -4543,6 +4552,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5116,6 +5127,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5528,6 +5548,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6528,6 +6556,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15650,6 +15682,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16238,6 +16277,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20766,3 +20809,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 42726fe3a6..628b9d10e2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index c330b21217..37615b6e85 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -294,9 +299,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -495,10 +500,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -526,7 +530,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -623,3 +627,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..636398baf1
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,58 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation
+ * having resolver */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index d38a22ddde..3ba83a25cd 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -96,4 +96,17 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
TimestampTz committs);
extern bool CanCreateFullTuple(Relation localrel,
LogicalRepTupleData *newtup);
+
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index c21486dbb4..802a7583d4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,5 +1,8 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | last_update_wins
@@ -9,9 +12,7 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(5 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
@@ -20,16 +21,14 @@ SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_differ | keep_local
@@ -42,7 +41,7 @@ select * from pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | keep_local
@@ -52,3 +51,235 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(5 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable_2;
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1_1 | delete_missing | error | t
+ ptntable_1_1 | insert_exists | remote_apply | t
+ ptntable_1_10 | delete_missing | error | t
+ ptntable_1_10 | insert_exists | remote_apply | t
+ ptntable_1_20 | delete_missing | error | t
+ ptntable_1_20 | insert_exists | remote_apply | t
+(8 rows)
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(8 rows)
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index f83d14b229..b4574d1df4 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,17 +1,19 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
@@ -19,10 +21,155 @@ SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
On Tue, Jul 9, 2024 at 3:09 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Jul 5, 2024 at 5:12 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Thank you Ajin for reporting the issue, This is now fixed with the
v4-0003 patch.Please find v5 patch-set. Changes are:
1) patch003:
Added test cases for all resolvers (034_conflict_resolver.pl).2) Patch004:
a) Emit error while resolving conflict if conflict resolver is default
'last_update_wins' but track_commit_timetsamp is not enabled.
b) Emit Warning during create and alter subscription when
'detect_conflict' is ON but 'track_commit_timetsamp' is not enabled.
c) Restrict start of pa worker if either max-clock-skew is configured
or conflict detection and resolution is enabled for a subscription.
d) Implement clock-skew delay/error when changes are applied from a
file (apply_spooled_messages).
e) Implement clock-skew delay while applying prepared changes (two
phase txns). The prepare-timestamp to be considered as base for
clock-skew handling as well as for last_update_win resolver.
<TODO: This needs to be analyzed and tested further to see if there is
any side effect of taking prepare-timestamp as base.>Thanks Ajin fo working on 1.
Thanks Nisha for working on 2a,2b.
Please find v6 patch-set. Changes are:
1) patch003:
1a) Improved log and restructured code around it.
1b) added test case for delete_differ.
2) patch004:
2a) Local and remote timestamps were logged incorrectly due to a bug,
corrected that.
2b) Added tests for last_update_wins.
2c) Added a cap on wait time; introduced a new GUC for this. Apply
worker will now error out without waiting if the computed wait exceeds
this GUC's value.
2d) Restricted enabling two_phase and detect_conflict together for a
subscription. This is because the time based resolvers may result in
data divergence for two phase commit transactions if prepare-timestamp
is used for comparison.
Thanks Nisha for working on 1a to 2b.
thanks
Shveta
Attachments:
v6-0003-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v6-0003-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From a104b11c86d5245a5595b5fb061fa6776d0bc704 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 15:46:58 +0530
Subject: [PATCH v6 3/5] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: remote_apply, keep_local, error
Each of the conflict type now has defined actions to manage data synchronization
between Publisher and subscriber databases.
---
src/backend/executor/execReplication.c | 79 ++-
src/backend/replication/logical/conflict.c | 205 ++++++-
src/backend/replication/logical/worker.c | 355 ++++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 14 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/029_on_error.pl | 7 +
.../subscription/t/034_conflict_resolver.pl | 579 ++++++++++++++++++
8 files changed, 1113 insertions(+), 132 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f01927a933..b18723e56b 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -566,7 +566,8 @@ retry:
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -602,21 +603,80 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ &apply_remote, NULL);
+
+ ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
conflictindexes, &conflict,
conflictindexes, false);
- /* Re-check all the unique indexes for potential conflicts */
+ /*
+ * Re-check all the unique indexes for potential conflicts.
+ *
+ * The Reason for doing the check again is:
+ *
+ * 1. If the remote change violated multiple unique constraints, the
+ * current resolution cannot solve it, so we still need to report this
+ * conflict after resolution.
+ *
+ * 2. If the local data is changed immediately after converting the
+ * insert to update but before updating the data, then the conflict
+ * can still happen, and we may need to report it again.
+ *
+ * XXX: Needs further review and discussion on usefulness of this
+ * repeated call.
+ */
foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
{
- TupleTableSlot *conflictslot;
/*
* Reports the conflict if any.
@@ -631,15 +691,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* cost of finding the tuple should be acceptable in this case.
*/
if (list_member_oid(recheckIndexes, uniqueidx) &&
- FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &(*conflictslot)))
{
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
- GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
- xmin, origin, committs, conflictslot);
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(CT_INSERT_EXISTS, CR_ERROR, rel, uniqueidx,
+ xmin, origin, committs, *conflictslot,
+ false);
}
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 93d9659752..574afe71f3 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -21,13 +21,17 @@
#include "access/table.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/timestamp.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -86,11 +90,13 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
- TupleTableSlot *conflictslot);
+ TupleTableSlot *conflictslot,
+ bool apply_remote);
/*
@@ -126,22 +132,32 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot,
+ bool apply_remote)
{
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
- errmsg("conflict %s detected on relation \"%s.%s\"",
+ errmsg("conflict %s detected on relation \"%s.%s\". Resolution: %s",
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
- RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
- localts, conflictslot));
+ RelationGetRelationName(localrel),
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
+ localts, conflictslot, apply_remote));
}
/*
@@ -176,13 +192,21 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot, bool apply_remote)
{
+ char *applymsg;
+
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
switch (type)
{
case CT_INSERT_EXISTS:
@@ -195,27 +219,43 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (resolver == CR_ERROR)
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists, %s", applymsg);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ else if (apply_remote)
+ return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
+ else
+ return errdetail("Did not find the row to be updated, %s",
+ applymsg);
case CT_DELETE_MISSING:
return errdetail("Did not find the row to be deleted.");
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
}
return 0; /* silence compiler warning */
@@ -329,6 +369,67 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
return type;
}
+/*
+ * Get the global conflict resolver configured for given conflict type
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache1(CONFLICTTYPE,
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(CONFLICTTYPE,
+ tuple, Anum_pg_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Execute Conflict Resolver Stmt
*
@@ -403,3 +504,47 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
table_close(pg_conflict, RowExclusiveLock);
}
+
+/*
+ * Find the global resolver for the given conflict type.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7e7de195ed..9859f0f2bd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2431,10 +2433,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2457,9 +2459,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2469,7 +2475,35 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2651,6 +2685,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2674,35 +2710,66 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
}
/* Cleanup. */
@@ -2815,6 +2882,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2836,24 +2905,42 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL);
- EvalPlanQualSetSlot(&epqstate, localslot);
+ ReportApplyConflict(CT_DELETE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
+
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
+
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+ }
}
/* Cleanup. */
@@ -2986,19 +3073,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3023,6 +3112,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3032,38 +3124,79 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
- InvalidRepOriginId, 0, NULL);
-
- return;
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL);
- /*
- * If conflict detection is enabled, check whether the local
- * tuple was modified by a different origin. If detected,
- * report the conflict.
- */
- if (MySubscription->detectconflict &&
- GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
- InvalidOid, localxmin, localorigin,
- localts, NULL);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3073,27 +3206,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3134,10 +3299,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3153,19 +3324,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..bc72b1f4fd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c6023e0914..b06c0f75a0 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -13,6 +13,8 @@
#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -74,10 +76,16 @@ typedef enum ConflictResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
-extern void ReportApplyConflict(int elevel, ConflictType type,
+extern void ReportApplyConflict(ConflictType type, ConflictResolver resolver,
Relation localrel, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot, bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup);
+extern bool CanCreateFullTuple(Relation localrel, LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..c746191601 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,10 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',"set conflict resolver 'error' for 'insert_exists'");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +181,9 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..0fb5ef9a86
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,579 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql('postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on);");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT conftype, confres FROM pg_conflict ORDER BY conftype"
+);
+is( $result, qq(delete_differ|remote_apply
+delete_missing|skip
+insert_exists|remote_apply
+update_differ|remote_apply
+update_missing|apply_or_skip),
+ "confirm that the default conflict resolvers are in place"
+);
+
+############################################
+# Test 'remote_apply' for 'insert_exists'
+############################################
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);"
+);
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'insert_exists';"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'delete_missing';"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);"
+);
+
+is($result, '', "delete from remote is applied");
+
+#########################################
+# Test 'keep_local' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'delete_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);"
+);
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'keep_local' for 'update_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);"
+);
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'update_differ';"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);"
+);
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update./, $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);"
+);
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres',
+ "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'skip' for 'update_missing';"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);"
+);
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'error' for 'update_missing';"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/, $log_offset);
+
+done_testing();
--
2.34.1
v6-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v6-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 691bd39b685677e2f316087c679eb442132d1a15 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 16 Jul 2024 13:54:23 +0530
Subject: [PATCH v6 4/5] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
This patch also implements last_update_wins resolver.
Since conflict resolution for two phase commit transactions
using prepare-timestamp can result in data divergence, this patch
also restricts enabling two_phase and detect_conflict together
for a subscription
---
src/backend/commands/subscriptioncmds.c | 45 +++++
src/backend/executor/execReplication.c | 2 +-
.../replication/logical/applyparallelworker.c | 26 ++-
src/backend/replication/logical/conflict.c | 142 ++++++++++++----
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 155 ++++++++++++++++--
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 +++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/catalog/pg_conflict.dat | 6 +-
src/include/replication/conflict.h | 6 +-
src/include/replication/logicalworker.h | 18 ++
src/include/replication/origin.h | 1 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
.../regress/expected/conflict_resolver.out | 16 +-
src/test/regress/expected/subscription.out | 2 +
src/test/subscription/t/029_on_error.pl | 52 ++++--
.../subscription/t/034_conflict_resolver.pl | 107 +++++++++++-
src/tools/pgindent/typedefs.list | 1 +
20 files changed, 556 insertions(+), 77 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 512b4273ae..7e3c7af160 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/xact.h"
@@ -459,6 +460,22 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Time based conflict resolution for two phase transactions can result in
+ * data divergence, so disallow enabling both together.
+ */
+ if (opts->detectconflict &&
+ IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ if (opts->twophase &&
+ IsSet(opts->specified_opts, SUBOPT_TWOPHASE_COMMIT))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "detect_conflict = true", "two_phase = true")));
+ }
}
/*
@@ -671,6 +688,13 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
#endif
+ /* Warn if detect_conflict is enabled and track_commit_timestamp is off */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
+
rel = table_open(SubscriptionRelationId, RowExclusiveLock);
/* Check if name is used */
@@ -1279,6 +1303,27 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ /*
+ * Time based conflict resolution for two phase
+ * transactions can result in data divergence, so disallow
+ * enabling it when two_phase is enabled.
+ */
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s for a subscription that has two_phase enabled",
+ "detect_conflict")));
+
+ /*
+ * Warn if detect_conflict is enabled and
+ * track_commit_timestamp is off.
+ */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
}
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index b18723e56b..8d93a965f2 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -630,7 +630,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool apply_remote = false;
GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ resolver = GetConflictResolver(*conflictslot, rel, CT_INSERT_EXISTS,
&apply_remote, NULL);
ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..10c7ca99df 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -312,6 +312,20 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Don't start a new parallel worker if user has either configured max
+ * clock skew or if conflict detection and resolution is ON. In both cases
+ * we need commit timestamp in the beginning.
+ *
+ * XXX: For conflict reolution case, see if we can reduce the scope of
+ * this restriction to only such cases where time-based resolvers are
+ * actually being used.
+ */
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ MySubscription->detectconflict)
+ return false;
+
+
return true;
}
@@ -696,9 +710,19 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured, thus it is okay to pass 0 as origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with conflict
+ * detection enabled, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 574afe71f3..a8fa470cf7 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -44,6 +44,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -69,11 +70,11 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -81,11 +82,11 @@ const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
* If this changes, change it in pg_conflict.dat as well.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+ [CT_DELETE_DIFFER] = CR_LAST_UPDATE_WINS
};
@@ -201,6 +202,12 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
TupleTableSlot *conflictslot, bool apply_remote)
{
char *applymsg;
+ int errdet = 0;
+ char *local_ts;
+ char *remote_ts;
+
+ local_ts = pstrdup(timestamptz_to_str(localts));
+ remote_ts = pstrdup(timestamptz_to_str(replorigin_session_origin_timestamp));
if (apply_remote)
applymsg = "applying the remote changes.";
@@ -222,43 +229,63 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
if (resolver == CR_ERROR)
{
if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx),
+ localorigin, localxmin, local_ts);
else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ errdet = errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
}
+ else if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Key already exists, %s. The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ applymsg, localorigin, local_ts,
+ replorigin_session_origin, remote_ts);
else
- return errdetail("Key already exists, %s", applymsg);
+ errdet = errdetail("Key already exists, %s", applymsg);
}
+ break;
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
case CT_UPDATE_MISSING:
if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
else if (apply_remote)
- return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
else
- return errdetail("Did not find the row to be updated, %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated, %s",
+ applymsg);
+ break;
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ errdet = errdetail("Did not find the row to be deleted.");
+ break;
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
}
- return 0; /* silence compiler warning */
+ pfree(local_ts);
+ pfree(remote_ts);
+
+ return errdet;
}
/*
@@ -366,6 +393,15 @@ validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -407,6 +443,42 @@ get_conflict_resolver_internal(ConflictType type)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -512,7 +584,8 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup)
{
ConflictResolver resolver;
@@ -521,6 +594,17 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9859f0f2bd..5f0af32e29 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,20 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -987,6 +1001,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1008,6 +1111,15 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
+ /*
+ * Capture the commit timestamp of the remote transaction
+ * for time based conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -1065,6 +1177,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1305,7 +1420,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2002,7 +2118,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2055,6 +2172,16 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
+ /*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /* Capture the timestamp (prepare or commit) of the remote transaction */
+ replorigin_session_origin_timestamp = origin_timestamp;
+
/*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
@@ -2160,7 +2287,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -2717,7 +2845,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
&apply_remote, NULL);
ReportApplyConflict(CT_UPDATE_DIFFER, resolver, localrel,
@@ -2753,7 +2881,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup);
ReportApplyConflict(CT_UPDATE_MISSING, resolver, localrel,
@@ -2906,7 +3034,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_DIFFER,
&apply_remote, NULL);
ReportApplyConflict(CT_DELETE_DIFFER, resolver, localrel,
@@ -2932,7 +3060,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL);
/* Resolver is set to skip, thus report the conflict and skip */
@@ -3130,7 +3258,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup);
ReportApplyConflict(CT_UPDATE_MISSING, resolver, partrel,
@@ -3162,7 +3290,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
&apply_remote, NULL);
ReportApplyConflict(CT_UPDATE_DIFFER, resolver, partrel,
@@ -4682,6 +4810,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4722,10 +4851,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37beeaae..a51f82169e 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 630ed0f162..8c843c522e 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -67,6 +67,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -492,6 +493,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3649,6 +3651,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4915,6 +4944,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97e92..f7a664a538 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
index 8d6183e3f2..1905151840 100644
--- a/src/include/catalog/pg_conflict.dat
+++ b/src/include/catalog/pg_conflict.dat
@@ -12,10 +12,10 @@
[
-{ conftype => 'insert_exists', confres => 'remote_apply' },
-{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'insert_exists', confres => 'last_update_wins' },
+{ conftype => 'update_differ', confres => 'last_update_wins' },
{ conftype => 'update_missing', confres => 'apply_or_skip' },
{ conftype => 'delete_missing', confres => 'skip' },
-{ conftype => 'delete_differ', confres => 'remote_apply' }
+{ conftype => 'delete_differ', confres => 'last_update_wins' }
]
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index b06c0f75a0..4e2853b5be 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -57,6 +57,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -83,7 +86,8 @@ extern void ReportApplyConflict(ConflictType type, ConflictResolver resolver,
TupleTableSlot *conflictslot, bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup);
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..7cb03062ac 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 515aefd519..dc9e067fac 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -267,7 +267,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index 53ecb430fa..c21486dbb4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,11 +1,11 @@
--check default global resolvers in system catalog
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+---------------
- delete_differ | remote_apply
+ conftype | confres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
update_missing | apply_or_skip
(5 rows)
@@ -43,11 +43,11 @@ RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
select * from pg_conflict order by conftype;
- conftype | confres
-----------------+----------------
+ conftype | confres
+----------------+------------------
delete_differ | keep_local
delete_missing | skip
- insert_exists | remote_apply
+ insert_exists | last_update_wins
update_differ | keep_local
update_missing | apply_or_error
(5 rows)
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 3d8b8c5d32..844a5dfff8 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -433,6 +433,8 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index c746191601..6c1eb7e5a9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -18,7 +18,7 @@ my $offset = 0;
# on the publisher.
sub test_skip_lsn
{
- my ($node_publisher, $node_subscriber, $nonconflict_data, $expected, $msg)
+ my ($node_publisher, $node_subscriber, $nonconflict_data, $expected, $msg, $conflict_detection)
= @_;
# Wait until a conflict occurs on the subscriber.
@@ -26,13 +26,25 @@ sub test_skip_lsn
"SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'"
);
+ my $lsn;
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
+
# Get the finish LSN of the error transaction, mapping the expected
# ERROR with its CONTEXT when retrieving this information.
- my $contents = slurp_file($node_subscriber->logfile, $offset);
- $contents =~
- qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
- or die "could not get error-LSN";
- my $lsn = $1;
+ if ($conflict_detection)
+ {
+ $contents =~
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
+ else
+ {
+ $contents =~
+ qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
# Set skip lsn.
$node_subscriber->safe_psql('postgres',
@@ -110,7 +122,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, detect_conflict = on)"
);
# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
@@ -147,7 +159,22 @@ INSERT INTO tbl VALUES (1, NULL);
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(2, NULL)", "2", "test skipping transaction");
+ "(2, NULL)", "2", "test skipping transaction", 1);
+
+# Cleanup before we start PREPARE AND COMMIT PREPARED tests
+$node_subscriber->safe_psql('postgres', "TRUNCATE tbl");
+$node_publisher->safe_psql('postgres', "TRUNCATE tbl");
+
+# Drop subscription and recreate with two_phase enabled
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'sub');
+
+$node_subscriber->safe_psql('postgres', "INSERT INTO tbl VALUES (1, NULL)");
# Test for PREPARE and COMMIT PREPARED. Insert the same data to tbl and
# PREPARE the transaction, raising an error. Then skip the transaction.
@@ -160,7 +187,7 @@ PREPARE TRANSACTION 'gtx';
COMMIT PREPARED 'gtx';
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(3, NULL)", "3", "test skipping prepare and commit prepared ");
+ "(2, NULL)", "2", "test skipping prepare and commit prepared ", 0);
# Test for STREAM COMMIT. Insert enough rows to tbl to exceed the 64kB
# limit, also raising an error on the subscriber during applying spooled
@@ -173,17 +200,14 @@ INSERT INTO tbl SELECT i, sha256(i::text::bytea) FROM generate_series(1, 10000)
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(4, sha256(4::text::bytea))",
- "4", "test skipping stream-commit");
+ "(3, sha256(4::text::bytea))",
+ "3", "test skipping stream-commit", 0);
$result = $node_subscriber->safe_psql('postgres',
"SELECT COUNT(*) FROM pg_prepared_xacts");
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
-# Reset conflict resolver for 'insert_exist' conflict type to default.
-$node_subscriber->safe_psql('postgres',"reset conflict resolver for 'insert_exists'");
-
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 0fb5ef9a86..3bde2750da 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -51,10 +51,10 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
my $result = $node_subscriber->safe_psql('postgres',
"SELECT conftype, confres FROM pg_conflict ORDER BY conftype"
);
-is( $result, qq(delete_differ|remote_apply
+is( $result, qq(delete_differ|last_update_wins
delete_missing|skip
-insert_exists|remote_apply
-update_differ|remote_apply
+insert_exists|last_update_wins
+update_differ|last_update_wins
update_missing|apply_or_skip),
"confirm that the default conflict resolvers are in place"
);
@@ -63,6 +63,11 @@ update_missing|apply_or_skip),
# Test 'remote_apply' for 'insert_exists'
############################################
+# Change CONFLICT RESOLVER of insert_exists to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'remote_apply' for 'insert_exists';"
+);
+
# Create local data on the subscriber
$node_subscriber->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
@@ -146,6 +151,34 @@ $node_subscriber->wait_for_log(
$node_subscriber->safe_psql('postgres',
"TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'last_update_wins' for 'insert_exists';"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);"
+);
+
+is($result, 'frompub', "remote data wins");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -203,14 +236,48 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
#########################################
-# Test 'remote_apply' for 'delete_differ'
+# Test 'last_update_wins' for 'delete_differ'
#########################################
+# Change CONFLICT RESOLVER of delete_differ to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'last_update_wins' for 'delete_differ';"
+);
+
# Insert data in the publisher
$node_publisher->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);"
+);
+
+is($result, '', "delete from remote wins");
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'remote_apply' for 'delete_differ';"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -303,9 +370,14 @@ $node_subscriber->safe_psql('postgres',
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
#########################################
-# Test 'remote_apply' for 'update_differ'
+# Test 'last_update_wins' for 'update_differ'
#########################################
+# Change CONFLICT RESOLVER of update_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "SET CONFLICT RESOLVER 'remote_apply' for 'update_differ';"
+);
+
# Insert data in the publisher
$node_publisher->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
@@ -331,6 +403,29 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/, $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);"
+);
+
+is($result, 'frompubnew2', "update from remote is kept");
+
#########################################
# Test 'keep_local' for 'update_differ'
#########################################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 1dfcc57fb1..ae3ee18764 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1567,6 +1567,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
v6-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v6-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 0cbd31d6ac288ecc98cc6572fd80f78a9fd210e4 Mon Sep 17 00:00:00 2001
From: Hou Zhijie <houzj.fnst@fujitsu.com>
Date: Fri, 31 May 2024 14:20:45 +0530
Subject: [PATCH v6 1/5] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 23 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 66 +++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 117 +++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 191 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 79 ++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 47 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 ++++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 801 insertions(+), 152 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index ccdd24312b..f078d6364b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1565,7 +1565,9 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped, but note that this scenario can be reported in the server log
+ if <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ is enabled.
</para>
<para>
@@ -1634,6 +1636,25 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row. Refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for other conflicts that will be logged when enabling <literal>detect_conflict</literal>.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 476f195622..5f6b83e415 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -228,8 +228,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-disable-on-error"><literal>disable_on_error</literal></link>,
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
- <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>, and
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>.
+ <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..caa523b9bd 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,72 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted is not found.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 16d83b3253..512b4273ae 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -70,8 +70,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -97,6 +98,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -159,6 +161,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -316,6 +320,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -603,7 +616,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -710,6 +724,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1148,7 +1164,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_STREAMING | SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1258,6 +1274,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..f01927a933 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,83 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +587,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +605,43 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, (conflict ? conflictindexes : NIL))
+ {
+ TupleTableSlot *conflictslot;
+
+ /*
+ * Reports the conflict if any.
+ *
+ * Here, we attempt to find the conflict tuple. This operation may
+ * seem redundant with the unique violation check of indexam, but
+ * since we perform this only when we are detecting conflict in
+ * logical replication and encountering potential conflicts with
+ * any unique index constraints (which should not be frequent), so
+ * it's ok. Moreover, upon detecting a conflict, we will report an
+ * ERROR and restart the logical replication, so the additional
+ * cost of finding the tuple should be acceptable in this case.
+ */
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, CT_INSERT_EXISTS, rel, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..b90e64b05b
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,191 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c0bda6269b..7e7de195ed 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2461,7 +2462,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2664,6 +2668,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2681,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2810,6 +2825,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2821,13 +2850,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2994,6 +3020,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3005,16 +3034,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index d453e224d9..219fac7e71 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..a9f521aaca
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,47 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 5c2f1ee517..3d8b8c5d32 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,10 +371,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
--fail - alter of two_phase option not supported.
@@ -383,10 +383,10 @@ ERROR: unrecognized subscription parameter: "two_phase"
-- but can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -396,10 +396,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -412,18 +412,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3e5ba4cb8c..a77b196704 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -290,6 +290,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..8c929c07c7 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4d7f9217c..2098ed7467 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v6-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchapplication/octet-stream; name=v6-0002-DDL-command-to-configure-Global-Conflict-Resolver.patchDownload
From 3540c8d1b24be2170842d06baa13f3ab40018be1 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 3 Jul 2024 10:23:58 +0530
Subject: [PATCH v6 2/5] DDL command to configure Global Conflict Resolvers
This patch adds new DDL commands to configure resolvers
at global level:
a) To set global resolver for a given conflict_type:
SET CONFLICT RESOLVER 'conflict_resolver' FOR 'conflict_type'
b) To reset to default:
RESET CONFLICT RESOLVER FOR 'conflict_type'
A new catalog table pg_conflict has been created to store
global conflict_type and conflict_resolver configurations given by
above DDL command. Initially, this catalog table holds default
configuration for resolvers which is overwritten by the ones given
by the user in 'SET CONFLICT RESOLVER' command.
'RESET CONFLICT RESOLVER' command will reset the resolver to default.
This patch provides support to configure resolvers for below conflict
types:
- insert_exists
- update_differ
- update_missing
- delete_missing
- delete_differ
---
src/backend/parser/gram.y | 42 +++-
src/backend/replication/logical/conflict.c | 218 +++++++++++++++++-
src/backend/tcop/utility.c | 15 ++
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 2 +
src/include/catalog/pg_conflict.dat | 21 ++
src/include/catalog/pg_conflict.h | 43 ++++
src/include/nodes/parsenodes.h | 12 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 38 ++-
src/include/tcop/cmdtaglist.h | 1 +
.../regress/expected/conflict_resolver.out | 54 +++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/conflict_resolver.sql | 28 +++
src/tools/pgindent/typedefs.list | 2 +
15 files changed, 476 insertions(+), 7 deletions(-)
create mode 100644 src/include/catalog/pg_conflict.dat
create mode 100644 src/include/catalog/pg_conflict.h
create mode 100644 src/test/regress/expected/conflict_resolver.out
create mode 100644 src/test/regress/sql/conflict_resolver.sql
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4c66..42726fe3a6 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -292,7 +292,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
AlterRoleStmt AlterRoleSetStmt AlterPolicyStmt AlterStatsStmt
AlterDefaultPrivilegesStmt DefACLAction
AnalyzeStmt CallStmt ClosePortalStmt ClusterStmt CommentStmt
- ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
+ ConflictResolverStmt ConstraintsSetStmt CopyStmt CreateAsStmt CreateCastStmt
CreateDomainStmt CreateExtensionStmt CreateGroupStmt CreateOpClassStmt
CreateOpFamilyStmt AlterOpFamilyStmt CreatePLangStmt
CreateSchemaStmt CreateSeqStmt CreateStmt CreateStatsStmt CreateTableSpaceStmt
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type conflict_resolver
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -772,7 +773,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -1039,6 +1040,7 @@ stmt:
| ClosePortalStmt
| ClusterStmt
| CommentStmt
+ | ConflictResolverStmt
| ConstraintsSetStmt
| CopyStmt
| CreateAmStmt
@@ -7266,7 +7268,41 @@ comment_text:
| NULL_P { $$ = NULL; }
;
+/*****************************************************************************
+ *
+ * SET CONFLICT RESOLVER <conflict_resolver> FOR <conflict_type>
+ * RESET CONFLICT RESOLVER FOR <conflict_type>
+ *
+ *****************************************************************************/
+
+ConflictResolverStmt:
+ SET CONFLICT RESOLVER conflict_resolver FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+
+ n->conflict_resolver = $4;
+ n->conflict_type = $6;
+ n->isReset = false;
+ $$ = (Node *) n;
+ }
+ | RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ ConflictResolverStmt *n = makeNode(ConflictResolverStmt);
+ n->conflict_type = $5;
+ n->isReset = true;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
+
+conflict_resolver:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
+ ;
/*****************************************************************************
*
* SECURITY LABEL [FOR <provider>] ON <object> IS <label>
@@ -17797,6 +17833,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18465,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b90e64b05b..93d9659752 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -1,13 +1,13 @@
/*-------------------------------------------------------------------------
* conflict.c
- * Functionality for detecting and logging conflicts.
+ * Functionality for detecting, logging and configuring conflicts.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/conflict.c
*
- * This file contains the code for detecting and logging conflicts on
+ * This file contains the code for detecting and handling conflicts on
* the subscriber during logical replication.
*-------------------------------------------------------------------------
*/
@@ -15,10 +15,19 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/genam.h"
+#include "access/htup_details.h"
+#include "access/skey.h"
+#include "access/table.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_conflict.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/timestamp.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -28,6 +37,54 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_DIFFER] = "delete_differ"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ * If this changes, change it in pg_conflict.dat as well.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -35,6 +92,7 @@ static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TimestampTz localts,
TupleTableSlot *conflictslot);
+
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
* with the provided local tuple.
@@ -189,3 +247,159 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Validate the conflict type and resolver.
+ *
+ * Return ConflictType enum value corresponding to given conflict_type.
+ */
+static ConflictType
+validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* If it is RESET statement, we are done */
+ if (isReset)
+ return type;
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ /*
+ * XXX: Shall we give supported resolvers list for given type in error
+ * hint?
+ */
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+}
+
+/*
+ * Execute Conflict Resolver Stmt
+ *
+ * Validate conflict_type and conflict_resolver given by the user
+ * and save the info in pg_conflict system catalog.
+ */
+void
+ExecConflictResolverStmt(ConflictResolverStmt *stmt)
+{
+ Relation pg_conflict;
+ Datum values[Natts_pg_conflict];
+ bool nulls[Natts_pg_conflict];
+ bool replaces[Natts_pg_conflict];
+ ScanKeyData keys[1];
+ SysScanDesc scan;
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ ConflictType type;
+
+ type = validate_conflict_type_and_resolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
+
+
+ /* Prepare to update a tuple in pg_conflict system catalog */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_conflict_conftype - 1] =
+ CStringGetTextDatum(stmt->conflict_type);
+
+ if (stmt->isReset)
+ {
+ /* Set resolver to default */
+ ConflictResolver resolver = ConflictTypeDefaultResolvers[type];
+
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(ConflictResolverNames[resolver]);
+ }
+ else
+ values[Anum_pg_conflict_confres - 1] =
+ CStringGetTextDatum(stmt->conflict_resolver);
+
+ /* Use the index to search for a matching old tuple */
+ ScanKeyInit(&keys[0],
+ Anum_pg_conflict_conftype,
+ BTEqualStrategyNumber, F_TEXTEQ,
+ values[Anum_pg_conflict_conftype - 1]);
+
+ pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+
+ scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
+ NULL, 1, keys);
+
+ oldtup = systable_getnext(scan);
+ if (HeapTupleIsValid(oldtup))
+ {
+ replaces[Anum_pg_conflict_confres - 1] = true;
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict, &oldtup->t_self, newtup);
+ }
+ else
+ Assert(oldtup); /* We must have a tuple for given
+ * conflict_type */
+
+ systable_endscan(scan);
+
+ if (newtup != NULL)
+ heap_freetuple(newtup);
+
+ table_close(pg_conflict, RowExclusiveLock);
+}
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index fa66b8017e..4ddf4aaff0 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -59,6 +59,7 @@
#include "miscadmin.h"
#include "parser/parse_utilcmd.h"
#include "postmaster/bgwriter.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "storage/fd.h"
#include "tcop/utility.h"
@@ -163,6 +164,7 @@ ClassifyUtilityCommandAsReadOnly(Node *parsetree)
case T_AlterUserMappingStmt:
case T_CommentStmt:
case T_CompositeTypeStmt:
+ case T_ConflictResolverStmt:
case T_CreateAmStmt:
case T_CreateCastStmt:
case T_CreateConversionStmt:
@@ -1062,6 +1064,11 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;
}
+ case T_ConflictResolverStmt:
+ /* no event triggers for conflict types and resolvers */
+ ExecConflictResolverStmt((ConflictResolverStmt *) parsetree);
+ break;
+
default:
/* All other statement types have event trigger support */
ProcessUtilitySlow(pstate, pstmt, queryString,
@@ -2659,6 +2666,10 @@ CreateCommandTag(Node *parsetree)
tag = CMDTAG_COMMENT;
break;
+ case T_ConflictResolverStmt:
+ tag = CMDTAG_CONFLICT_RESOLVER;
+ break;
+
case T_SecLabelStmt:
tag = CMDTAG_SECURITY_LABEL;
break;
@@ -3334,6 +3345,10 @@ GetCommandLogLevel(Node *parsetree)
lev = LOGSTMT_DDL;
break;
+ case T_ConflictResolverStmt:
+ lev = LOGSTMT_DDL;
+ break;
+
case T_SecLabelStmt:
lev = LOGSTMT_DDL;
break;
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..8619d73e5a 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
@@ -97,6 +98,7 @@ POSTGRES_BKI_DATA = \
pg_cast.dat \
pg_class.dat \
pg_collation.dat \
+ pg_conflict.dat \
pg_conversion.dat \
pg_database.dat \
pg_language.dat \
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..4d6732a303 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
@@ -81,6 +82,7 @@ bki_data = [
'pg_cast.dat',
'pg_class.dat',
'pg_collation.dat',
+ 'pg_conflict.dat',
'pg_conversion.dat',
'pg_database.dat',
'pg_language.dat',
diff --git a/src/include/catalog/pg_conflict.dat b/src/include/catalog/pg_conflict.dat
new file mode 100644
index 0000000000..8d6183e3f2
--- /dev/null
+++ b/src/include/catalog/pg_conflict.dat
@@ -0,0 +1,21 @@
+#----------------------------------------------------------------------
+#
+# pg_conflict.dat
+# Initial contents of the pg_conflict system catalog.
+#
+# Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/include/catalog/pg_conflict.dat
+#
+#----------------------------------------------------------------------
+
+[
+
+{ conftype => 'insert_exists', confres => 'remote_apply' },
+{ conftype => 'update_differ', confres => 'remote_apply' },
+{ conftype => 'update_missing', confres => 'apply_or_skip' },
+{ conftype => 'delete_missing', confres => 'skip' },
+{ conftype => 'delete_differ', confres => 'remote_apply' }
+
+]
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
new file mode 100644
index 0000000000..e3fe3e6d30
--- /dev/null
+++ b/src/include/catalog/pg_conflict.h
@@ -0,0 +1,43 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict.h
+ * definition of the "global conflict detection" system catalog
+ * (pg_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_H
+#define PG_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_d.h"
+
+/* ----------------
+ * pg_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_conflict
+ * ----------------
+ */
+CATALOG(pg_conflict,8688,ConflictResRelationId)
+{
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict;
+
+DECLARE_TOAST(pg_conflict, 8689, 8690);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_type_index, 8691, ConflictTypeIndexId, pg_conflict, btree(conftype text_ops));
+
+MAKE_SYSCACHE(CONFLICTTYPE, pg_conflict_type_index, 16);
+
+#endif /* PG_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..1d41169d5e 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3262,6 +3262,18 @@ typedef struct CommentStmt
char *comment; /* Comment to insert, or NULL to remove */
} CommentStmt;
+/* ----------------------
+ * CONFLICT RESOLVER Statement
+ * ----------------------
+ */
+typedef struct ConflictResolverStmt
+{
+ NodeTag type;
+ char *conflict_type;
+ char *conflict_resolver;
+ bool isReset;
+} ConflictResolverStmt;
+
/* ----------------------
* SECURITY LABEL Statement
* ----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index a9f521aaca..c6023e0914 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "access/xlogdefs.h"
+#include "catalog/objectaddress.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
#include "utils/relcache.h"
@@ -36,6 +37,41 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_DIFFER
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -43,5 +79,5 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-
+extern void ExecConflictResolverStmt(ConflictResolverStmt *stmt);
#endif
diff --git a/src/include/tcop/cmdtaglist.h b/src/include/tcop/cmdtaglist.h
index 7fdcec6dd9..562478e232 100644
--- a/src/include/tcop/cmdtaglist.h
+++ b/src/include/tcop/cmdtaglist.h
@@ -80,6 +80,7 @@ PG_CMDTAG(CMDTAG_CLUSTER, "CLUSTER", false, false, false)
PG_CMDTAG(CMDTAG_COMMENT, "COMMENT", true, false, false)
PG_CMDTAG(CMDTAG_COMMIT, "COMMIT", false, false, false)
PG_CMDTAG(CMDTAG_COMMIT_PREPARED, "COMMIT PREPARED", false, false, false)
+PG_CMDTAG(CMDTAG_CONFLICT_RESOLVER, "CONFLICT RESOLVER", false, false, false)
PG_CMDTAG(CMDTAG_COPY, "COPY", false, false, true)
PG_CMDTAG(CMDTAG_COPY_FROM, "COPY FROM", false, false, false)
PG_CMDTAG(CMDTAG_CREATE_ACCESS_METHOD, "CREATE ACCESS METHOD", true, false, false)
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
new file mode 100644
index 0000000000..53ecb430fa
--- /dev/null
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -0,0 +1,54 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_missing | apply_or_skip
+(5 rows)
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+ERROR: aaaa is not a valid conflict type
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+ERROR: bbbbb is not a valid conflict resolver
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+ERROR: ct is not a valid conflict type
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | error
+ insert_exists | keep_local
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
+ conftype | confres
+----------------+----------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | keep_local
+ update_missing | apply_or_error
+(5 rows)
+
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 2429ec2bba..76ea2d359b 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -48,7 +48,7 @@ test: create_index create_index_spgist create_view index_including index_includi
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_sql create_cast constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
+test: create_aggregate create_function_sql create_cast conflict_resolver constraints triggers select inherit typed_table vacuum drop_if_exists updatable_views roleattributes create_am hash_func errors infinite_recurse
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
new file mode 100644
index 0000000000..f83d14b229
--- /dev/null
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -0,0 +1,28 @@
+--check default global resolvers in system catalog
+select * from pg_conflict order by conftype;
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with invalid names
+--
+SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
+SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
+SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
+RESET CONFLICT RESOLVER for 'ct'; -- fail
+
+--
+-- Test of SET/RESET CONFLICT RESOLVER with valid names
+--
+SET CONFLICT RESOLVER 'error' for 'delete_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
+SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
+SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
+SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
+
+--check new resolvers are saved
+select * from pg_conflict order by conftype;
+
+RESET CONFLICT RESOLVER for 'delete_missing';
+RESET CONFLICT RESOLVER for 'insert_exists';
+
+--check resolvers are reset to default for delete_missing and insert_exists
+select * from pg_conflict order by conftype;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2098ed7467..1dfcc57fb1 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,8 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
+ConflictResolverStmt
ConflictType
ConnCacheEntry
ConnCacheKey
--
2.34.1
v6-0005-Configure-table-level-conflict-resolvers.patchapplication/octet-stream; name=v6-0005-Configure-table-level-conflict-resolvers.patchDownload
From ed9ec14fa145a4fedf0571bd1098264d89657f20 Mon Sep 17 00:00:00 2001
From: Shveta Malik <shveta.malik@gmail.com>
Date: Wed, 17 Jul 2024 10:13:38 +0530
Subject: [PATCH v6 5/5] Configure table level conflict resolvers
This patch provides support for configuring table level conflict
resolvers using ALTER TABLE cmd.
Syntax to SET resolvers:
ALTER TABLE <name> SET CONFLICT RESOLVER <resolver1> on <conflict_type1>,
SET CONFLICT RESOLVER <resolver2> on <conflict_type2>, ...;
A new catalog table pg_conflict_rel has been created to store
table-level conflict_type and conflict_resolver configurations given by
above DDL command.
Syntax to RESET resolvers:
ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type1>,
RESET CONFLICT RESOLVER on <conflict_type2>, ...;
Above RESET command will remove entry for that particular conflict_type for
the given table from pg_conflict_rel catalog table.
---
src/backend/catalog/dependency.c | 6 +
src/backend/catalog/objectaddress.c | 30 ++
src/backend/commands/tablecmds.c | 70 ++++
src/backend/parser/gram.y | 30 ++
src/backend/replication/logical/conflict.c | 330 +++++++++++++++++-
src/include/catalog/Makefile | 4 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_conflict.h | 2 +-
src/include/catalog/pg_conflict_rel.h | 58 +++
src/include/nodes/parsenodes.h | 2 +
src/include/replication/conflict.h | 12 +
.../regress/expected/conflict_resolver.out | 245 ++++++++++++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/sql/conflict_resolver.sql | 161 ++++++++-
14 files changed, 928 insertions(+), 24 deletions(-)
create mode 100644 src/include/catalog/pg_conflict_rel.h
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 0489cbabcb..9dd78d9094 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -31,6 +31,7 @@
#include "catalog/pg_auth_members.h"
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
@@ -79,6 +80,7 @@
#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parsetree.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteRemove.h"
#include "storage/lmgr.h"
#include "utils/fmgroids.h"
@@ -1444,6 +1446,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemovePublicationById(object->objectId);
break;
+ case ConflictRelRelationId:
+ RemoveTableConflictById(object->objectId);
+ break;
+
case CastRelationId:
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 85a7b7e641..1456eedf46 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -30,6 +30,7 @@
#include "catalog/pg_cast.h"
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
+#include "catalog/pg_conflict_rel.h"
#include "catalog/pg_conversion.h"
#include "catalog/pg_database.h"
#include "catalog/pg_default_acl.h"
@@ -3005,6 +3006,35 @@ getObjectDescription(const ObjectAddress *object, bool missing_ok)
break;
}
+ case ConflictRelRelationId:
+ {
+ HeapTuple confTup;
+ Datum typeDatum;
+ Datum resDatum;
+ char *contype;
+ char *conres;
+
+ confTup = SearchSysCache1(CONFLICTRELOID,
+ ObjectIdGetDatum(object->objectId));
+ if (!HeapTupleIsValid(confTup))
+ {
+ if (!missing_ok)
+ elog(ERROR, "cache lookup failed for table conflict %u",
+ object->objectId);
+ break;
+ }
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrtype);
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, confTup,
+ Anum_pg_conflict_rel_confrres);
+ contype = TextDatumGetCString(typeDatum);
+ conres = TextDatumGetCString(resDatum);
+ ReleaseSysCache(confTup);
+ appendStringInfo(&buffer, _("conflict_resolver %s on conflict_type %s"),
+ conres, contype);
+ break;
+ }
case ConstraintRelationId:
{
HeapTuple conTup;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 721d24783b..4ebd80950a 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -83,6 +83,7 @@
#include "partitioning/partbounds.h"
#include "partitioning/partdesc.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "rewrite/rewriteDefine.h"
#include "rewrite/rewriteHandler.h"
#include "rewrite/rewriteManip.h"
@@ -662,6 +663,11 @@ static void ATExecSplitPartition(List **wqueue, AlteredTableInfo *tab,
AlterTableUtilityContext *context);
static void ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
PartitionCmd *cmd, AlterTableUtilityContext *context);
+static void ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+static void
+ ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode);
/* ----------------------------------------------------------------
* DefineRelation
@@ -1245,6 +1251,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
*/
CloneForeignKeyConstraints(NULL, parent, rel);
+ /* Inherit conflict resolvers configuration from parent. */
+ InheritTableConflictResolvers(rel, parent);
+
table_close(parent, NoLock);
}
@@ -4543,6 +4552,8 @@ AlterTableGetLockLevel(List *cmds)
case AT_SetExpression:
case AT_DropExpression:
case AT_SetCompression:
+ case AT_SetConflictResolver:
+ case AT_ResetConflictResolver:
cmd_lockmode = AccessExclusiveLock;
break;
@@ -5116,6 +5127,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
/* No command-specific prep needed */
pass = AT_PASS_MISC;
break;
+ case AT_ResetConflictResolver: /* RESET CONFLICT RESOLVER */
+ case AT_SetConflictResolver: /* SET CONFLICT RESOLVER */
+ ATSimplePermissions(cmd->subtype, rel, ATT_TABLE);
+ /* Recursion occurs during execution phase */
+ /* No command-specific prep needed except saving recurse flag */
+ if (recurse)
+ cmd->recurse = true;
+ pass = AT_PASS_MISC;
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -5528,6 +5548,14 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab,
ATExecMergePartitions(wqueue, tab, rel, (PartitionCmd *) cmd->def,
context);
break;
+ case AT_SetConflictResolver:
+ ATExecSetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
+ case AT_ResetConflictResolver:
+ ATExecResetConflictResolver(rel, (ConflictResolverStmt *) cmd->def,
+ cmd->recurse, false, lockmode);
+ break;
default: /* oops */
elog(ERROR, "unrecognized alter table type: %d",
(int) cmd->subtype);
@@ -6528,6 +6556,10 @@ alter_table_type_to_string(AlterTableType cmdtype)
return "ALTER COLUMN ... DROP IDENTITY";
case AT_ReAddStatistics:
return NULL; /* not real grammar */
+ case AT_SetConflictResolver:
+ return "SET CONFLICT RESOLVER";
+ case AT_ResetConflictResolver:
+ return "RESET CONFLICT RESOLVER";
}
return NULL;
@@ -15650,6 +15682,13 @@ CreateInheritance(Relation child_rel, Relation parent_rel, bool ispartition)
/* Match up the constraints and bump coninhcount as needed */
MergeConstraintsIntoExisting(child_rel, parent_rel);
+ /*
+ * Inherit resolvers configuration from parent if not explicitly set for
+ * child partition.
+ */
+ if (ispartition)
+ InheritTableConflictResolvers(child_rel, parent_rel);
+
/*
* OK, it looks valid. Make the catalog entries that show inheritance.
*/
@@ -16238,6 +16277,10 @@ RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached)
systable_endscan(scan);
table_close(catalogRelation, RowExclusiveLock);
+ /* Find inherited conflict resolvers and disinherit them */
+ if (is_partitioning)
+ ResetResolversInheritance(child_rel);
+
drop_parent_dependency(RelationGetRelid(child_rel),
RelationRelationId,
RelationGetRelid(parent_rel),
@@ -20775,3 +20818,30 @@ ATExecMergePartitions(List **wqueue, AlteredTableInfo *tab, Relation rel,
/* Keep the lock until commit. */
table_close(newPartRel, NoLock);
}
+
+/*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER ...
+ */
+static void
+ATExecSetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, false);
+
+ SetTableConflictResolver(NULL, rel, stmt->conflict_type,
+ stmt->conflict_resolver,
+ recurse, recursing, lockmode);
+}
+
+/*
+ * ALTER TABLE <name> RESET CONFLICT RESOLVER ...
+ */
+static void
+ATExecResetConflictResolver(Relation rel, ConflictResolverStmt *stmt,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ ValidateConflictTypeAndResolver(stmt->conflict_type, stmt->conflict_resolver, true);
+
+ ResetTableConflictResolver(rel, stmt->conflict_type,
+ recurse, recursing, lockmode);
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 42726fe3a6..628b9d10e2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -3015,6 +3015,36 @@ alter_table_cmd:
n->subtype = AT_GenericOptions;
n->def = (Node *) $1;
+ $$ = (Node *) n;
+ }
+ /*
+ * ALTER TABLE <name> SET CONFLICT RESOLVER <conflict_resolver>
+ * on <conflict_type>
+ */
+ | SET CONFLICT RESOLVER conflict_resolver ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_resolver = $4;
+ c->conflict_type = $6;
+
+ n->subtype = AT_SetConflictResolver;
+ n->def = (Node *) c;
+
+ $$ = (Node *) n;
+ }
+ /* ALTER TABLE <name> RESET CONFLICT RESOLVER on <conflict_type> */
+ | RESET CONFLICT RESOLVER ON conflict_type
+ {
+ AlterTableCmd *n = makeNode(AlterTableCmd);
+ ConflictResolverStmt *c = makeNode(ConflictResolverStmt);
+
+ c->conflict_type = $5;
+
+ n->subtype = AT_ResetConflictResolver;
+ n->def = (Node *) c;
+
$$ = (Node *) n;
}
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index a8fa470cf7..7ec9fd6dd8 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -19,8 +19,11 @@
#include "access/htup_details.h"
#include "access/skey.h"
#include "access/table.h"
+#include "catalog/dependency.h"
#include "catalog/indexing.h"
#include "catalog/pg_conflict.h"
+#include "catalog/pg_conflict_rel.h"
+#include "catalog/pg_inherits.h"
#include "executor/executor.h"
#include "replication/conflict.h"
#include "replication/logicalproto.h"
@@ -28,8 +31,10 @@
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
#include "utils/timestamp.h"
#include "utils/syscache.h"
@@ -320,9 +325,9 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
*
* Return ConflictType enum value corresponding to given conflict_type.
*/
-static ConflictType
-validate_conflict_type_and_resolver(char *conflict_type, char *conflict_resolver,
- bool isReset)
+ConflictType
+ValidateConflictTypeAndResolver(char *conflict_type, char *conflict_resolver,
+ bool isReset)
{
ConflictType type;
ConflictResolver resolver;
@@ -521,10 +526,9 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
HeapTuple newtup = NULL;
ConflictType type;
- type = validate_conflict_type_and_resolver(stmt->conflict_type,
- stmt->conflict_resolver,
- stmt->isReset);
-
+ type = ValidateConflictTypeAndResolver(stmt->conflict_type,
+ stmt->conflict_resolver,
+ stmt->isReset);
/* Prepare to update a tuple in pg_conflict system catalog */
memset(values, 0, sizeof(values));
@@ -552,7 +556,7 @@ ExecConflictResolverStmt(ConflictResolverStmt *stmt)
BTEqualStrategyNumber, F_TEXTEQ,
values[Anum_pg_conflict_conftype - 1]);
- pg_conflict = table_open(ConflictResRelationId, RowExclusiveLock);
+ pg_conflict = table_open(ConflictRelationId, RowExclusiveLock);
scan = systable_beginscan(pg_conflict, ConflictTypeIndexId, true,
NULL, 1, keys);
@@ -632,3 +636,313 @@ GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
return resolver;
}
+
+/*
+ * Update the table level conflict resolver for a conflict_type in
+ * pg_conflict_rel system catalog.
+ */
+void
+SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type, char *conflict_resolver,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Relation pg_conflict_rel;
+ Datum values[Natts_pg_conflict_rel];
+ bool nulls[Natts_pg_conflict_rel];
+ bool replaces[Natts_pg_conflict_rel];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Oid relid = rel->rd_id;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ if (!pg_conf_rel)
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+ else
+ pg_conflict_rel = pg_conf_rel;
+
+ values[Anum_pg_conflict_rel_confrrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_conflict_rel_confrtype - 1] = CStringGetTextDatum(conflict_type);
+ values[Anum_pg_conflict_rel_confrres - 1] = CStringGetTextDatum(conflict_resolver);
+ values[Anum_pg_conflict_rel_confrinherited - 1] = BoolGetDatum(recursing);
+
+ oldtup = SearchSysCache2(CONFLICTRELTYPE,
+ values[Anum_pg_conflict_rel_confrrelid - 1],
+ values[Anum_pg_conflict_rel_confrtype - 1]);
+ if (HeapTupleIsValid(oldtup))
+ {
+ Form_pg_conflict_rel confForm = (Form_pg_conflict_rel) GETSTRUCT(oldtup);
+
+ /*
+ * Update resolver for recursing=false cases.
+ *
+ * recursing=true indicates it is a child parittion table and if it
+ * already has a resolver set then overwrite it only if it is an
+ * inherited resolver. We should not overwrite non-inherited resolvers
+ * for child partition tables during recursion.
+ */
+ if (!recursing || (recursing && confForm->confrinherited))
+ {
+ replaces[Anum_pg_conflict_rel_confrres - 1] = true;
+ replaces[Anum_pg_conflict_rel_confrinherited - 1] = !recursing;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_conflict_rel),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_conflict_rel, &oldtup->t_self, newtup);
+ }
+ else
+ {
+ /*
+ * If we did not update resolver for this child table, do not
+ * update for child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+ /* If we didn't find an old tuple, insert a new one */
+ else
+ {
+ ObjectAddress myself,
+ referenced;
+ Oid conflict_oid;
+
+ conflict_oid = GetNewOidWithIndex(pg_conflict_rel, ConflictRelOidIndexId,
+ Anum_pg_conflict_rel_oid);
+ values[Anum_pg_conflict_rel_oid - 1] = ObjectIdGetDatum(conflict_oid);
+
+ newtup = heap_form_tuple(RelationGetDescr(pg_conflict_rel),
+ values, nulls);
+ CatalogTupleInsert(pg_conflict_rel, newtup);
+
+ /* Add dependency on the relation */
+ ObjectAddressSet(myself, ConflictRelRelationId, conflict_oid);
+ ObjectAddressSet(referenced, RelationRelationId, relid);
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
+ }
+
+ if (HeapTupleIsValid(newtup))
+ heap_freetuple(newtup);
+
+ if (!pg_conf_rel)
+ table_close(pg_conflict_rel, RowExclusiveLock);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ SetTableConflictResolver(NULL, childrel, conflict_type, conflict_resolver, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Reset table level conflict_type configuration.
+ *
+ * Removes table's resolver configuration for given conflict_type from
+ * pg_conflict_rel system catalog.
+ */
+void
+ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode)
+{
+ Oid relid = rel->rd_id;
+ HeapTuple tup;
+ ObjectAddress confobj;
+ Form_pg_conflict_rel confForm;
+
+ tup = SearchSysCache2(CONFLICTRELTYPE,
+ ObjectIdGetDatum(relid),
+ CStringGetTextDatum(conflict_type));
+
+ /* Nothing to reset */
+ if (!HeapTupleIsValid(tup))
+ return;
+
+ confForm = (Form_pg_conflict_rel) GETSTRUCT(tup);
+
+ if (confForm->confrinherited && !recursing)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TABLE_DEFINITION),
+ errmsg("cannot reset inherited resolver for conflict type \"%s\" of relation \"%s\"",
+ conflict_type, RelationGetRelationName(rel))));
+
+ /*
+ * While resetting resolver on parent table, reset resolver for child
+ * partition table only if it is inherited one.
+ *
+ * OTOH, if user has invoked RESET directly on child partition table,
+ * ensure we reset only non-inherited resolvers. Inherited resolvers can
+ * not be reset alone on child table, RESET has to come through parent
+ * table.
+ */
+ if ((recursing && confForm->confrinherited) ||
+ (!recursing && !confForm->confrinherited))
+ {
+ ObjectAddressSet(confobj, ConflictRelRelationId, confForm->oid);
+ performDeletion(&confobj, DROP_CASCADE, 0);
+ }
+ else
+ {
+ /*
+ * If we did not reset resolver for this child table, do not reset for
+ * child's child tables as well.
+ */
+ recurse = false;
+ }
+
+ ReleaseSysCache(tup);
+
+ if (recurse && (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE))
+ {
+ List *children;
+ ListCell *lc;
+
+ children =
+ find_inheritance_children(RelationGetRelid(rel), lockmode);
+
+ foreach(lc, children)
+ {
+ Relation childrel;
+
+ /* find_inheritance_children already got lock */
+ childrel = table_open(lfirst_oid(lc), NoLock);
+ ResetTableConflictResolver(childrel, conflict_type, recurse, true, lockmode);
+ table_close(childrel, NoLock);
+ }
+ }
+}
+
+/*
+ * Inherit conflict resolvers from parent paritioned table.
+ *
+ * If child partition table has resolvers set already for some conflict
+ * types, do not overwrite those, inherit rest from parent.
+ *
+ * Used during attach partition operation.
+ */
+void
+InheritTableConflictResolvers(Relation child_rel, Relation parent_rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc parent_scan;
+ ScanKeyData parent_key;
+ HeapTuple parent_tuple;
+ Oid parent_relid = RelationGetRelid(parent_rel);
+
+ Assert(parent_rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&parent_key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(parent_relid));
+
+ parent_scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &parent_key);
+
+ while (HeapTupleIsValid(parent_tuple = systable_getnext(parent_scan)))
+ {
+ Datum typeDatum;
+ Datum resDatum;
+ char *parent_conftype;
+ char *parent_confres;
+
+ typeDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrtype);
+ parent_conftype = TextDatumGetCString(typeDatum);
+
+ resDatum = SysCacheGetAttrNotNull(CONFLICTRELOID, parent_tuple,
+ Anum_pg_conflict_rel_confrres);
+ parent_confres = TextDatumGetCString(resDatum);
+
+ SetTableConflictResolver(pg_conflict_rel, child_rel, parent_conftype, parent_confres, true, true, AccessExclusiveLock);
+ }
+
+ systable_endscan(parent_scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Reset inheritance of conflict resolvers.
+ *
+ * Used during detach partition operation. Detach partition
+ * will not remove inherited resolvers but will mark them
+ * as non inherited.
+ */
+void
+ResetResolversInheritance(Relation rel)
+{
+ Relation pg_conflict_rel;
+ SysScanDesc scan;
+ ScanKeyData key;
+ HeapTuple tuple;
+ Oid relid = RelationGetRelid(rel);
+
+ Assert(rel->rd_rel->relispartition);
+
+ pg_conflict_rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ ScanKeyInit(&key,
+ Anum_pg_conflict_rel_confrrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+
+ scan = systable_beginscan(pg_conflict_rel, InvalidOid, false, NULL, 1, &key);
+
+ while (HeapTupleIsValid(tuple = systable_getnext(scan)))
+ {
+ Form_pg_conflict_rel conf =
+ (Form_pg_conflict_rel) GETSTRUCT(tuple);
+
+ if (conf->confrinherited)
+ {
+ HeapTuple copy_tuple = heap_copytuple(tuple);
+ Form_pg_conflict_rel copy_conf =
+ (Form_pg_conflict_rel) GETSTRUCT(copy_tuple);
+
+ copy_conf->confrinherited = false;
+ CatalogTupleUpdate(pg_conflict_rel, ©_tuple->t_self, copy_tuple);
+ heap_freetuple(copy_tuple);
+
+ }
+ }
+
+ systable_endscan(scan);
+ table_close(pg_conflict_rel, RowExclusiveLock);
+}
+
+/*
+ * Remove the conflict resolver configuration by table conflict oid.
+ */
+void
+RemoveTableConflictById(Oid confid)
+{
+ Relation rel;
+ HeapTuple tup;
+
+ rel = table_open(ConflictRelRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(CONFLICTRELOID, ObjectIdGetDatum(confid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for table conflict %u", confid);
+
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 8619d73e5a..e192375fa0 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -82,7 +82,9 @@ CATALOG_HEADERS := \
pg_publication_rel.h \
pg_subscription.h \
pg_subscription_rel.h \
- pg_conflict.h
+ pg_conflict.h \
+ pg_conflict_rel.h
+
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index 4d6732a303..28924f5df8 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -70,6 +70,7 @@ catalog_headers = [
'pg_subscription.h',
'pg_subscription_rel.h',
'pg_conflict.h',
+ 'pg_conflict_rel.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_conflict.h b/src/include/catalog/pg_conflict.h
index e3fe3e6d30..c080a0d0be 100644
--- a/src/include/catalog/pg_conflict.h
+++ b/src/include/catalog/pg_conflict.h
@@ -26,7 +26,7 @@
* typedef struct FormData_pg_conflict
* ----------------
*/
-CATALOG(pg_conflict,8688,ConflictResRelationId)
+CATALOG(pg_conflict,8688,ConflictRelationId)
{
#ifdef CATALOG_VARLEN /* variable-length fields start here */
text conftype BKI_FORCE_NOT_NULL; /* conflict type */
diff --git a/src/include/catalog/pg_conflict_rel.h b/src/include/catalog/pg_conflict_rel.h
new file mode 100644
index 0000000000..636398baf1
--- /dev/null
+++ b/src/include/catalog/pg_conflict_rel.h
@@ -0,0 +1,58 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_conflict_rel.h
+ * definition of the "table level conflict detection" system
+ * catalog (pg_conflict_rel)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_conflict_rel.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_CONFLICT_REL_H
+#define PG_CONFLICT_REL_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_conflict_rel_d.h"
+
+/* ----------------
+ * pg_conflict_rel definition. cpp turns this into
+ * typedef struct FormData_pg_conflict_rel
+ * ----------------
+ */
+CATALOG(pg_conflict_rel,8881,ConflictRelRelationId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confrrelid BKI_LOOKUP(pg_class); /* Oid of the relation
+ * having resolver */
+ bool confrinherited; /* Is this an inherited configuration */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_conflict_rel;
+
+/* ----------------
+ * Form_pg_conflict_rel corresponds to a pointer to a row with
+ * the format of pg_conflict_rel relation.
+ * ----------------
+ */
+typedef FormData_pg_conflict_rel * Form_pg_conflict_rel;
+
+DECLARE_TOAST(pg_conflict_rel, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_conflict_rel_oid_index, 8884, ConflictRelOidIndexId, pg_conflict_rel, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_conflict_rel_type_index, 8885, ConflictRelTypeIndexId, pg_conflict_rel, btree(confrrelid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(CONFLICTRELOID, pg_conflict_rel_oid_index, 256);
+MAKE_SYSCACHE(CONFLICTRELTYPE, pg_conflict_rel_type_index, 256);
+
+#endif /* PG_CONFLICT_REL_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1d41169d5e..7677cf023b 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2419,6 +2419,8 @@ typedef enum AlterTableType
AT_SetIdentity, /* SET identity column options */
AT_DropIdentity, /* DROP IDENTITY */
AT_ReAddStatistics, /* internal to commands/tablecmds.c */
+ AT_SetConflictResolver, /* SET CONFLICT RESOLVER */
+ AT_ResetConflictResolver, /* RESET CONFLICT RESOLVER */
} AlterTableType;
typedef struct ReplicaIdentityStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 4e2853b5be..207c4cd9a4 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -92,4 +92,16 @@ extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
bool *apply_remote,
LogicalRepTupleData *newtup);
extern bool CanCreateFullTuple(Relation localrel, LogicalRepTupleData *newtup);
+extern ConflictType ValidateConflictTypeAndResolver(char *conflict_type,
+ char *conflict_resolver,
+ bool isReset);
+extern void SetTableConflictResolver(Relation pg_conf_rel, Relation rel, char *conflict_type,
+ char *conflict_resolver, bool recurse,
+ bool recursing, LOCKMODE lockmode);
+extern void ResetTableConflictResolver(Relation rel, char *conflict_type,
+ bool recurse, bool recursing, LOCKMODE lockmode);
+extern void RemoveTableConflictById(Oid confid);
+extern void InheritTableConflictResolvers(Relation child_rel, Relation parent_rel);
+extern void ResetResolversInheritance(Relation rel);
+
#endif
diff --git a/src/test/regress/expected/conflict_resolver.out b/src/test/regress/expected/conflict_resolver.out
index c21486dbb4..802a7583d4 100644
--- a/src/test/regress/expected/conflict_resolver.out
+++ b/src/test/regress/expected/conflict_resolver.out
@@ -1,5 +1,8 @@
+--
+-- Test for configuration of global resolvers
+--
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | last_update_wins
@@ -9,9 +12,7 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_skip
(5 rows)
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
ERROR: aaaa is not a valid conflict type
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
@@ -20,16 +21,14 @@ SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
ERROR: remote_apply is not a valid conflict resolver for conflict type delete_missing
RESET CONFLICT RESOLVER for 'ct'; -- fail
ERROR: ct is not a valid conflict type
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+----------------
delete_differ | keep_local
@@ -42,7 +41,7 @@ select * from pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
conftype | confres
----------------+------------------
delete_differ | keep_local
@@ -52,3 +51,235 @@ select * from pg_conflict order by conftype;
update_missing | apply_or_error
(5 rows)
+--
+-- Test for configuration of table level resolvers
+--
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | error | t
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2_1 | delete_missing | error | t
+ ptntable_2_1 | insert_exists | keep_local | t
+(8 rows)
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | keep_local | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | keep_local | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | keep_local | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | keep_local | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+ERROR: cannot reset inherited resolver for conflict type "insert_exists" of relation "ptntable_2"
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+(8 rows)
+
+DROP TABLE ptntable_2;
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1_1 | delete_missing | error | t
+ ptntable_1_1 | insert_exists | remote_apply | t
+ ptntable_1_10 | delete_missing | error | t
+ ptntable_1_10 | insert_exists | remote_apply | t
+ ptntable_1_20 | delete_missing | error | t
+ ptntable_1_20 | insert_exists | remote_apply | t
+(8 rows)
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+(4 rows)
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(8 rows)
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | t
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+-- Test for ALTER TABLE..DETACH PARTITION
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+--------------+----------------+--------------+----------------
+ ptntable | delete_missing | error | f
+ ptntable | insert_exists | remote_apply | f
+ ptntable_1 | delete_missing | error | t
+ ptntable_1 | insert_exists | remote_apply | t
+ ptntable_2 | delete_missing | skip | f
+ ptntable_2 | insert_exists | remote_apply | f
+ ptntable_2 | update_differ | remote_apply | f
+ ptntable_2_1 | delete_missing | skip | t
+ ptntable_2_1 | insert_exists | remote_apply | t
+ ptntable_2_1 | update_differ | remote_apply | t
+(10 rows)
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+ relname | confrtype | confrres | confrinherited
+---------+-----------+----------+----------------
+(0 rows)
+
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..8caf852600 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_conflict_rel {confrrelid} => pg_class {oid}
diff --git a/src/test/regress/sql/conflict_resolver.sql b/src/test/regress/sql/conflict_resolver.sql
index f83d14b229..b4574d1df4 100644
--- a/src/test/regress/sql/conflict_resolver.sql
+++ b/src/test/regress/sql/conflict_resolver.sql
@@ -1,17 +1,19 @@
+--
+-- Test for configuration of global resolvers
+--
+
--check default global resolvers in system catalog
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
---
-- Test of SET/RESET CONFLICT RESOLVER with invalid names
---
+
SET CONFLICT RESOLVER 'keep_local' for 'aaaa'; -- fail
SET CONFLICT RESOLVER 'bbbbb' for 'delete_missing'; -- fail
SET CONFLICT RESOLVER 'remote_apply' for 'delete_missing'; -- fail
RESET CONFLICT RESOLVER for 'ct'; -- fail
---
-- Test of SET/RESET CONFLICT RESOLVER with valid names
---
+
SET CONFLICT RESOLVER 'error' for 'delete_missing';
SET CONFLICT RESOLVER 'keep_local' for 'insert_exists';
SET CONFLICT RESOLVER 'keep_local' for 'update_differ';
@@ -19,10 +21,155 @@ SET CONFLICT RESOLVER 'apply_or_error' for 'update_missing';
SET CONFLICT RESOLVER 'keep_local' for 'delete_differ';
--check new resolvers are saved
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
RESET CONFLICT RESOLVER for 'delete_missing';
RESET CONFLICT RESOLVER for 'insert_exists';
--check resolvers are reset to default for delete_missing and insert_exists
-select * from pg_conflict order by conftype;
+SELECT * FROM pg_conflict order by conftype;
+
+--
+-- Test for configuration of table level resolvers
+--
+
+
+-- Test for ALTER TABLE..SET/RESET CONFLICT RESOLVER
+
+CREATE TABLE ptntable (
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_1 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-02-01') TO ('2006-03-01');
+
+CREATE TABLE ptntable_2 PARTITION OF ptntable
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01')
+ PARTITION BY RANGE (logdate);
+
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'keep_local' ON 'insert_exists';
+
+--Expect 8 entries. 2 for parent and 2 for each child partition (inherited
+--resolvers)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'skip' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ';
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The resolver for 'delete_missing' changed from 'error' to 'skip'
+--The 'update_differ'configuration added
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Set resolvers on parent table again. It should not overwrite non inherited
+--entries for ptntable_2 and corresponding entries for ptntable_2_1
+ALTER TABLE ptntable SET CONFLICT RESOLVER 'error' ON 'delete_missing',
+ SET CONFLICT RESOLVER 'remote_apply' ON 'insert_exists';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Reset of inherited entry ON child alone should result in error
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'insert_exists';
+
+--Reset of non-inherited entry ON child alone should suceed
+ALTER TABLE ptntable_2 RESET CONFLICT RESOLVER ON 'update_differ';
+
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+
+--Expect entries removed for both ptntable_2 and its child ptntable_2_1
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..SPLIT PARTITION and ALTER TABLE..MERGE PARTITIONS
+
+ALTER TABLE ptntable SPLIT PARTITION ptntable_1 INTO
+ (PARTITION ptntable_1_1 FOR VALUES FROM ('2006-02-01') TO ('2006-02-10'),
+ PARTITION ptntable_1_10 FOR VALUES FROM ('2006-02-10') TO ('2006-02-20'),
+ PARTITION ptntable_1_20 FOR VALUES FROM ('2006-02-20') TO ('2006-03-01'));
+
+--Expect split partitions to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+ALTER TABLE ptntable MERGE PARTITIONS (ptntable_1_1, ptntable_1_10, ptntable_1_20)
+ INTO ptntable_1;
+
+--Expect merged partition to inherit resolvers FROM parent
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..ATTACH PARTITION and CREATE TABLE..PARTITION OF
+
+CREATE TABLE ptntable_2(
+ city_id int not null,
+ logdate date not null,
+ peaktemp int,
+ unitsales int
+) PARTITION BY RANGE (logdate);
+
+ALTER TABLE ptntable_2 SET CONFLICT RESOLVER 'remote_apply' ON 'update_differ',
+ SET CONFLICT RESOLVER 'skip' ON 'delete_missing';
+
+--Expect ptntable_2_1 to inherit resolvers of ptntable_2
+CREATE TABLE ptntable_2_1 PARTITION OF ptntable_2
+ FOR VALUES FROM ('2006-03-20') TO ('2006-04-01');
+
+--Expect 4 new entries for ptntable_2 and its child ptntable_2_1
+--Entried for ptntable_2_1 should be marked as inherited.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+--Attach ptntable_2 to ptntable now
+ALTER TABLE ptntable ATTACH PARTITION ptntable_2
+ FOR VALUES FROM ('2006-03-01') TO ('2006-04-01');
+
+--For both ptntable_2 and ptntable_2_1, expect:
+--The 'insert_exists' configuration inherited
+--Resolver for 'delete_missing' not overwritten
+--Resolver for 'update_differ' retained.
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+
+-- Test for ALTER TABLE..DETACH PARTITION
+
+ALTER TABLE ptntable DETACH PARTITION ptntable_2;
+
+--All resolvers of ptntable_2 should be marked as non-inherited
+--All resolvers for ptntable_2_1's should still be marked as inherited (as
+--they are still inherited from ptntable_2)
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
+
+DROP TABLE ptntable_2;
+DROP TABLE ptntable;
+
+--Expect no resolvers
+SELECT relname, confrtype, confrres, confrinherited FROM pg_class pc,
+ (SELECT confrrelid, confrtype, confrres, confrinherited FROM pg_conflict_rel)conf
+ WHERE conf.confrrelid= pc.oid order by relname, confrtype;
--
2.34.1
On Wed, Jul 17, 2024 at 4:01 PM shveta malik <shveta.malik@gmail.com> wrote:
Please find v6 patch-set. Changes are:
Please find v7 patch-set, the changes are:
Patch 0001 - Reflects v5 of Conflict Detection patch in [1]/messages/by-id/OS0PR01MB57166C2566E00676649CF48B94AC2@OS0PR01MB5716.jpnprd01.prod.outlook.com.
Patch 0002:
a) Removed global CONFLICT RESOLVER syntax and logic.
b) Added new syntax for creating CONFLICT RESOLVERs at the subscription
level.
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 =
resolver3, ...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 =
resolver3, ...);
Patch 0003 - Supports subscription-level resolvers for conflict resolution.
Patch 0004 - Modified last_update_win related test cases to reflect the new
syntax.
Patch 0005 - Dropped for the time being; will rebase and post in the next
version.
Thanks to Shveta for design discussions and thanks to Nisha for helping in
rebasing the patch and helping in testing and stabilizing the patch by
providing comments off-list.
[1]: /messages/by-id/OS0PR01MB57166C2566E00676649CF48B94AC2@OS0PR01MB5716.jpnprd01.prod.outlook.com
/messages/by-id/OS0PR01MB57166C2566E00676649CF48B94AC2@OS0PR01MB5716.jpnprd01.prod.outlook.com
Attachments:
v7-0002-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchapplication/octet-stream; name=v7-0002-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchDownload
From e6651190f22de06689313475c04cc6f2c195f8d5 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 25 Jul 2024 00:12:40 -0400
Subject: [PATCH v7 2/4] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
---
src/backend/commands/subscriptioncmds.c | 110 +++++++
src/backend/parser/gram.y | 25 +-
src/backend/replication/logical/conflict.c | 310 ++++++++++++++++++
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 56 ++++
src/include/nodes/parsenodes.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 69 ++++
src/test/regress/sql/subscription.sql | 35 ++
12 files changed, 659 insertions(+), 4 deletions(-)
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b2bc095c71..0e40a2226d 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -28,6 +28,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +38,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -452,6 +454,46 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
}
+/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+ ConflictType type;
+ bool valid = false;
+
+ if (!stmtresolvers)
+ return;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(resolvers[type].conflict_type, defel->defname) == 0)
+ {
+ resolvers[type].resolver = defGetString(defel);
+ validate_conflict_type_and_resolver(resolvers[type].conflict_type,
+ resolvers[type].resolver);
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", defel->defname)));
+
+ /* reset the flag for the next conflict type */
+ valid = false;
+ }
+}
+
/*
* Add publication names from the list to a string.
*/
@@ -596,6 +638,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+ bool skip_conflict_resolvers = false;
/*
* Parse and check options.
@@ -611,6 +655,33 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ if (!opts.detectconflict)
+ {
+ /*
+ * If conflict resolvers are set but detect_conflict is not enabled,
+ * issue warning and ignore resolvers.
+ */
+ if (stmt->resolvers)
+ ereport(WARNING,
+ (errmsg("Ignoring given CONFLICT RESOLVERS as detect_conflict is not enabled.")));
+ skip_conflict_resolvers = true;
+ }
+ else
+ {
+ /* If conflict resolvers are not set, use default values */
+ if (!stmt->resolvers)
+ {
+ ereport(WARNING,
+ (errmsg("Will use default resolvers configuration as detect_conflict is ON but resolvers are not given")));
+ }
+
+ /*
+ * Parse and check conflict resolvers. Initialize with default values
+ */
+ SetDefaultResolvers(conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ }
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -742,6 +813,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ if (!skip_conflict_resolvers)
+ SetSubConflictResolver(subid, conflictResolvers);
+
/*
* Connect to remote side to execute requested commands and fetch table
* info.
@@ -1374,6 +1449,18 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
{
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+
+ if (!opts.detectconflict)
+ RemoveSubscriptionConflictBySubid(subid);
+ else
+ {
+ ereport(WARNING,
+ (errmsg("Using default conflict resolvers")));
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers);
+ }
+
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
@@ -1604,6 +1691,26 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /* make sure that detect_conflict is enabled, else throw error */
+ if (!sub->detectconflict)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set conflict resolvers when detect_conflict is not enabled")));
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1855,6 +1962,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4c66..a690b46c6a 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,7 +426,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -772,7 +772,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8824,6 +8824,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10758,7 +10763,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10766,6 +10771,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10872,6 +10878,17 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+
;
/*****************************************************************************
@@ -17797,6 +17814,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18446,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b7dc73cfce..104a8ba3cd 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,9 +15,23 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/rel.h"
const char *const ConflictTypeNames[] = {
@@ -30,6 +44,55 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_DIFFER] = "delete_differ"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -192,3 +255,250 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver.
+ */
+void
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *CTR = NULL;
+ List *res = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ CTR = palloc(sizeof(ConflictTypeResolver));
+ CTR->conflict_type = defel->defname;
+ CTR->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+
+ res = lappend(res, CTR);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolver for a conflict type in
+ * pg_subscription_conflict system catalog
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ ConflictType type;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[type].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[type].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ {
+ CatalogTupleDelete(rel, &tup->t_self);
+ }
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..ad82f69288
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,56 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTOID, pg_subscription_conflict_oid_index, 256);
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTSUBOID, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..2e1eefc3a7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4203,6 +4203,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4215,6 +4216,7 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4225,6 +4227,7 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834cf4..21e5cb162d 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 3a7260d3c1..91ab4908b1 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -39,6 +39,47 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_DIFFER
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -46,5 +87,13 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictBySubid(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern void validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index cc9337ce73..1bff69e1d9 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -430,6 +430,7 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: Will use default resolvers configuration as detect_conflict is ON but resolvers are not given
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
@@ -439,6 +440,18 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
(1 row)
+-- confirm that the default conflict resolvers have been set
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | apply_or_skip
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
\dRs+
List of subscriptions
@@ -447,6 +460,62 @@ ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
+-- confirm that the conflict resolvers have been dropped
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = foo);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- creating subscription with detect_conflict = false should not create conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- creating subscription with detect_conflict = false but conflict resolvers specified
+-- will result in conflict resolvers being ignored
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false) CONFLICT RESOLVER (insert_exists = 'keep_local');
+WARNING: Ignoring given CONFLICT RESOLVERS as detect_conflict is not enabled.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+-- setting detect_conflict to true will set default conflict resolvers
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+WARNING: Using default conflict resolvers
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | apply_or_skip
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- let's do some tests with pg_create_subscription rather than superuser
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5c740fd263..d5fb082b09 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -295,10 +295,45 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
\dRs+
+-- confirm that the default conflict resolvers have been set
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
\dRs+
+-- confirm that the conflict resolvers have been dropped
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- creating subscription with detect_conflict = false should not create conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- creating subscription with detect_conflict = false but conflict resolvers specified
+-- will result in conflict resolvers being ignored
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false) CONFLICT RESOLVER (insert_exists = 'keep_local');
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- setting detect_conflict to true will set default conflict resolvers
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--
2.34.1
v7-0003-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v7-0003-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From ac4879d814fc85bf59fd8a66e1d7b01f6d67d62c Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 25 Jul 2024 13:44:17 +0530
Subject: [PATCH v7 3/4] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: remote_apply, keep_local, error
---
src/backend/executor/execReplication.c | 56 +-
src/backend/replication/logical/conflict.c | 207 ++++++-
src/backend/replication/logical/worker.c | 366 ++++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 13 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/029_on_error.pl | 9 +
.../subscription/t/034_conflict_resolver.pl | 576 ++++++++++++++++++
8 files changed, 1104 insertions(+), 129 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 0680bc86fd..fb96bb7005 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -590,8 +590,9 @@ ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
- xmin, origin, committs, conflictslot);
+ ReportApplyConflict(type, CR_ERROR, resultRelInfo->ri_RelationDesc,
+ uniqueidx, xmin, origin, committs,
+ conflictslot, false);
}
}
}
@@ -604,7 +605,8 @@ ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -640,11 +642,55 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ &apply_remote, NULL, subid);
+
+ ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 104a8ba3cd..50f12a0efd 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -26,13 +26,17 @@
#include "catalog/pg_subscription_conflict_d.h"
#include "catalog/pg_inherits.h"
#include "commands/defrem.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -94,11 +98,13 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
- TupleTableSlot *conflictslot);
+ TupleTableSlot *conflictslot,
+ bool apply_remote);
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
@@ -133,22 +139,32 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot,
+ bool apply_remote)
{
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
- errmsg("conflict %s detected on relation \"%s.%s\"",
+ errmsg("conflict %s detected on relation \"%s.%s\". Resolution: %s",
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
- RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
- localts, conflictslot));
+ RelationGetRelationName(localrel),
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
+ localts, conflictslot, apply_remote));
}
/*
@@ -183,13 +199,21 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot, bool apply_remote)
{
+ char *applymsg;
+
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
switch (type)
{
case CT_INSERT_EXISTS:
@@ -203,27 +227,43 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (resolver == CR_ERROR)
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists, %s", applymsg);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ else if (apply_remote)
+ return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
+ else
+ return errdetail("Did not find the row to be updated, %s",
+ applymsg);
case CT_DELETE_MISSING:
return errdetail("Did not find the row to be deleted.");
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
}
return 0; /* silence compiler warning */
@@ -339,6 +379,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
@@ -502,3 +605,47 @@ RemoveSubscriptionConflictBySubid(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fc3f80e8b0..36172b2460 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2428,10 +2430,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2454,9 +2456,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2466,7 +2472,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MyLogicalRepWorker->subid);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL, InvalidOid);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2648,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2671,38 +2708,71 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- if (MySubscription->detectconflict)
- InitConflictIndexes(relinfo);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
}
/* Cleanup. */
@@ -2815,6 +2885,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2836,24 +2908,44 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_DELETE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
+
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+ }
}
/* Cleanup. */
@@ -2986,19 +3078,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3023,6 +3117,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3032,38 +3129,81 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
- InvalidRepOriginId, 0, NULL);
-
- return;
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
- /*
- * If conflict detection is enabled, check whether the local
- * tuple was modified by a different origin. If detected,
- * report the conflict.
- */
- if (MySubscription->detectconflict &&
- GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
- InvalidOid, localxmin, localorigin,
- localts, NULL);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3073,27 +3213,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3134,10 +3306,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3153,19 +3331,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752ea3..7d4a698c83 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 91ab4908b1..8ce20c5bb4 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -12,6 +12,8 @@
#include "access/xlogdefs.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -82,10 +84,11 @@ typedef struct ConflictTypeResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
-extern void ReportApplyConflict(int elevel, ConflictType type,
+extern void ReportApplyConflict(ConflictType type, ConflictResolver resolver,
Relation localrel, Oid conflictidx,
TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ TimestampTz localts, TupleTableSlot *conflictslot,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -95,5 +98,11 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern void validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
+extern bool CanCreateFullTuple(Relation localrel, LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6cd9..e6f07fac03 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,11 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=error)");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +182,10 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=remote_apply)");
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..58751c5c24
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,576 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on);");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_differ|remote_apply
+delete_missing|skip
+insert_exists|remote_apply
+update_differ|remote_apply
+update_exists|remote_apply
+update_missing|apply_or_skip),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'remote_apply' for 'insert_exists'
+############################################
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+#########################################
+# Test 'keep_local' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v7-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v7-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 65db17b5a7857f8878d398275bbe3c14325ca5bd Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Wed, 24 Jul 2024 23:58:36 -0400
Subject: [PATCH v7 1/4] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_exists: The updated row value violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 +
doc/src/sgml/logical-replication.sgml | 21 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 84 +++++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 +++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 142 +++++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 194 ++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 84 +++++++--
src/bin/pg_dump/pg_dump.c | 17 +-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 50 +++++
src/test/regress/expected/subscription.out | 176 ++++++++++--------
src/test/regress/sql/subscription.sql | 15 ++
src/test/subscription/t/001_rep_changes.pl | 15 +-
src/test/subscription/t/013_partition.pl | 68 ++++---
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 850 insertions(+), 155 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae1b2..b042a5a94a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8035,6 +8035,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index a23a3d57e2..d236d8530c 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1580,7 +1580,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped. Please refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling <literal>detect_conflict</literal>.
</para>
<para>
@@ -1649,6 +1650,24 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d007..dfbe25b59e 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -235,8 +235,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
<link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
- <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>.
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>,
+ <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d9421..eb51805e81 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,90 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc9159f2..5a423f4fb0 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9a47..d084bfc48a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe55c..b2bc095c71 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -71,8 +71,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -98,6 +99,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -162,6 +164,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -307,6 +311,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -594,7 +607,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -701,6 +715,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1196,7 +1212,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1356,6 +1372,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b3654c..ef522778a2 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd577..0680bc86fd 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -480,6 +481,121 @@ retry:
return found;
}
+/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Re-check all the unique indexes in 'recheckIndexes' to see if there are
+ * potential conflicts with the tuple in 'slot'.
+ *
+ * This function is invoked after inserting or updating a tuple that detected
+ * potential conflict tuples. It attempts to find the tuple that conflicts with
+ * the provided tuple. This operation may seem redundant with the unique
+ * violation check of indexam, but since we call this function only when we are
+ * detecting conflict in logical replication and encountering potential
+ * conflicts with any unique index constraints (which should not be frequent),
+ * so it's ok. Moreover, upon detecting a conflict, we will report an ERROR and
+ * restart the logical replication, so the additional cost of finding the tuple
+ * should be acceptable.
+ */
+static void
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -509,6 +625,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +643,17 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
@@ -577,6 +702,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
{
List *recheckIndexes = NIL;
TU_UpdateIndexes update_indexes;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -593,12 +720,19 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, true, false,
- NULL, NIL,
+ slot, estate, true,
+ conflictindexes,
+ &conflict, conflictindexes,
(update_indexes == TU_Summarizing));
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_UPDATE_EXISTS,
+ recheckIndexes, slot);
+
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
NULL, NULL,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eeff1c..1e08bbbd4e 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000000..b7dc73cfce
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,194 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_EXISTS] = "update_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ case CT_UPDATE_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a6de..3d36249d8a 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5fe85..fc3f80e8b0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2458,7 +2459,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2646,7 +2650,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2661,6 +2665,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2668,6 +2686,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
@@ -2678,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2807,6 +2825,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2818,13 +2850,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2991,6 +3020,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3002,16 +3034,28 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
+ /*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
/*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b8b1888bd3..829f9b3e88 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e5870a9..bbd7cbeff6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f234c..fef1ad0d70 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 891face1b6..086e135f65 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3363,9 +3364,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec4a2..17daf11dc7 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000000..3a7260d3c1
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..cc9337ce73 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +409,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..5c740fd263 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -287,6 +287,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981962..78c0307165 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 29580525a9..7a66a06b51 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4b5b..496a3c6cd9 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f008..8c929c07c7 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -139,6 +144,44 @@ is($result, qq(),
'Remote data originating from another node (not the publisher) is not replicated when origin parameter is none'
);
+###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4d7f9217c..2098ed7467 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
2.34.1
v7-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v7-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 576b65f0e775f048ae7afddf70fa77bec5bdb399 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 25 Jul 2024 13:56:38 +0530
Subject: [PATCH v7 4/4] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
This patch also implements last_update_wins resolver.
Since conflict resolution for two phase commit transactions
using prepare-timestamp can result in data divergence, this patch
also restricts enabling two_phase and detect_conflict together
for a subscription.
---
src/backend/commands/subscriptioncmds.c | 45 +++++
src/backend/executor/execReplication.c | 2 +-
.../replication/logical/applyparallelworker.c | 26 ++-
src/backend/replication/logical/conflict.c | 146 +++++++++++++----
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 155 ++++++++++++++++--
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 +++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/conflict.h | 6 +-
src/include/replication/logicalworker.h | 18 ++
src/include/replication/origin.h | 1 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
src/test/regress/expected/subscription.out | 28 ++--
src/test/subscription/t/029_on_error.pl | 53 ++++--
.../subscription/t/034_conflict_resolver.pl | 116 +++++++++++--
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 566 insertions(+), 85 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 0e40a2226d..8f9c474431 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -452,6 +453,22 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Time based conflict resolution for two phase transactions can result in
+ * data divergence, so disallow enabling both together.
+ */
+ if (opts->detectconflict &&
+ IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ if (opts->twophase &&
+ IsSet(opts->specified_opts, SUBOPT_TWOPHASE_COMMIT))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "detect_conflict = true", "two_phase = true")));
+ }
}
/*
@@ -733,6 +750,13 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
#endif
+ /* Warn if detect_conflict is enabled and track_commit_timestamp is off */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
+
rel = table_open(SubscriptionRelationId, RowExclusiveLock);
/* Check if name is used */
@@ -1464,6 +1488,27 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ /*
+ * Time based conflict resolution for two phase
+ * transactions can result in data divergence, so disallow
+ * enabling it when two_phase is enabled.
+ */
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s for a subscription that has two_phase enabled",
+ "detect_conflict")));
+
+ /*
+ * Warn if detect_conflict is enabled and
+ * track_commit_timestamp is off.
+ */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
}
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index fb96bb7005..93ba364319 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -669,7 +669,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool apply_remote = false;
GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ resolver = GetConflictResolver(*conflictslot, rel, CT_INSERT_EXISTS,
&apply_remote, NULL, subid);
ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..10c7ca99df 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -312,6 +312,20 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Don't start a new parallel worker if user has either configured max
+ * clock skew or if conflict detection and resolution is ON. In both cases
+ * we need commit timestamp in the beginning.
+ *
+ * XXX: For conflict reolution case, see if we can reduce the scope of
+ * this restriction to only such cases where time-based resolvers are
+ * actually being used.
+ */
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ MySubscription->detectconflict)
+ return false;
+
+
return true;
}
@@ -696,9 +710,19 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured, thus it is okay to pass 0 as origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with conflict
+ * detection enabled, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 50f12a0efd..91252da578 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -51,6 +51,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -76,24 +77,24 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
* Default conflict resolver for each conflict type.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+ [CT_DELETE_DIFFER] = CR_LAST_UPDATE_WINS
};
@@ -208,6 +209,12 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
TupleTableSlot *conflictslot, bool apply_remote)
{
char *applymsg;
+ int errdet = 0;
+ char *local_ts;
+ char *remote_ts;
+
+ local_ts = pstrdup(timestamptz_to_str(localts));
+ remote_ts = pstrdup(timestamptz_to_str(replorigin_session_origin_timestamp));
if (apply_remote)
applymsg = "applying the remote changes.";
@@ -230,43 +237,63 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
if (resolver == CR_ERROR)
{
if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx),
+ localorigin, localxmin, local_ts);
else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ errdet = errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
}
+ else if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Key already exists, %s. The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ applymsg, localorigin, local_ts,
+ replorigin_session_origin, remote_ts);
else
- return errdetail("Key already exists, %s", applymsg);
+ errdet = errdetail("Key already exists, %s", applymsg);
}
+ break;
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
case CT_UPDATE_MISSING:
if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
else if (apply_remote)
- return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
else
- return errdetail("Did not find the row to be updated, %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated, %s",
+ applymsg);
+ break;
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ errdet = errdetail("Did not find the row to be deleted.");
+ break;
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
}
- return 0; /* silence compiler warning */
+ pfree(local_ts);
+ pfree(remote_ts);
+
+ return errdet;
}
/*
@@ -377,6 +404,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
conflict_resolver,
conflict_type));
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
+
}
/*
@@ -419,6 +455,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -613,7 +685,8 @@ RemoveSubscriptionConflictBySubid(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
@@ -622,6 +695,17 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 36172b2460..5b3b89ebc4 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,20 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -984,6 +998,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1005,6 +1108,15 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -1062,6 +1174,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1302,7 +1417,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -1999,7 +2115,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2052,6 +2169,16 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
+ /*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /* Capture the timestamp (prepare or commit) of the remote transaction */
+ replorigin_session_origin_timestamp = origin_timestamp;
+
/*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
@@ -2157,7 +2284,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -2715,7 +2843,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -2755,7 +2883,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MyLogicalRepWorker->subid);
@@ -2909,7 +3037,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -2936,7 +3064,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -3135,7 +3263,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MyLogicalRepWorker->subid);
@@ -3168,7 +3296,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -4671,6 +4799,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4711,10 +4840,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37beeaae..a51f82169e 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index f6fcdebb03..c8c8a991e2 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3639,6 +3641,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4905,6 +4934,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97e92..f7a664a538 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 8ce20c5bb4..5aecefda8a 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -59,6 +59,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -98,7 +101,8 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern void validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..7cb03062ac 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261d7e..95b2a5286d 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 1bff69e1d9..ac15c30cdd 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -431,6 +431,8 @@ ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
WARNING: Will use default resolvers configuration as detect_conflict is ON but resolvers are not given
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
@@ -442,13 +444,13 @@ HINT: To initiate replication, you must manually create the replication slot, e
-- confirm that the default conflict resolvers have been set
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+---------------
- delete_differ | remote_apply
+ confrtype | confrres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
- update_exists | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | apply_or_skip
(6 rows)
@@ -499,14 +501,16 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
-- setting detect_conflict to true will set default conflict resolvers
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
WARNING: Using default conflict resolvers
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+---------------
- delete_differ | remote_apply
+ confrtype | confrres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
- update_exists | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | apply_or_skip
(6 rows)
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index e6f07fac03..9620a4bb1a 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -18,7 +18,8 @@ my $offset = 0;
# on the publisher.
sub test_skip_lsn
{
- my ($node_publisher, $node_subscriber, $nonconflict_data, $expected, $msg)
+ my ($node_publisher, $node_subscriber, $nonconflict_data, $expected,
+ $msg, $conflict_detection)
= @_;
# Wait until a conflict occurs on the subscriber.
@@ -26,13 +27,25 @@ sub test_skip_lsn
"SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'"
);
+ my $lsn;
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
+
# Get the finish LSN of the error transaction, mapping the expected
# ERROR with its CONTEXT when retrieving this information.
- my $contents = slurp_file($node_subscriber->logfile, $offset);
- $contents =~
- qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
- or die "could not get error-LSN";
- my $lsn = $1;
+ if ($conflict_detection)
+ {
+ $contents =~
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
+ else
+ {
+ $contents =~
+ qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
# Set skip lsn.
$node_subscriber->safe_psql('postgres',
@@ -110,7 +123,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, detect_conflict = on)"
);
# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
@@ -148,7 +161,22 @@ INSERT INTO tbl VALUES (1, NULL);
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(2, NULL)", "2", "test skipping transaction");
+ "(2, NULL)", "2", "test skipping transaction", 1);
+
+# Cleanup before we start PREPARE AND COMMIT PREPARED tests
+$node_subscriber->safe_psql('postgres', "TRUNCATE tbl");
+$node_publisher->safe_psql('postgres', "TRUNCATE tbl");
+
+# Drop subscription and recreate with two_phase enabled
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'sub');
+
+$node_subscriber->safe_psql('postgres', "INSERT INTO tbl VALUES (1, NULL)");
# Test for PREPARE and COMMIT PREPARED. Insert the same data to tbl and
# PREPARE the transaction, raising an error. Then skip the transaction.
@@ -161,7 +189,7 @@ PREPARE TRANSACTION 'gtx';
COMMIT PREPARED 'gtx';
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(3, NULL)", "3", "test skipping prepare and commit prepared ");
+ "(2, NULL)", "2", "test skipping prepare and commit prepared ", 0);
# Test for STREAM COMMIT. Insert enough rows to tbl to exceed the 64kB
# limit, also raising an error on the subscriber during applying spooled
@@ -174,17 +202,14 @@ INSERT INTO tbl SELECT i, sha256(i::text::bytea) FROM generate_series(1, 10000)
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(4, sha256(4::text::bytea))",
- "4", "test skipping stream-commit");
+ "(3, sha256(4::text::bytea))",
+ "3", "test skipping stream-commit", 0);
$result = $node_subscriber->safe_psql('postgres',
"SELECT COUNT(*) FROM pg_prepared_xacts");
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
-# Reset conflict resolver for 'insert_exist' conflict type to default.
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=remote_apply)");
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 58751c5c24..e9618094d0 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -54,11 +54,11 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
my $result = $node_subscriber->safe_psql('postgres',
"SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
);
-is( $result, qq(delete_differ|remote_apply
+is( $result, qq(delete_differ|last_update_wins
delete_missing|skip
-insert_exists|remote_apply
-update_differ|remote_apply
-update_exists|remote_apply
+insert_exists|last_update_wins
+update_differ|last_update_wins
+update_exists|last_update_wins
update_missing|apply_or_skip),
"confirm that the default conflict resolvers are in place");
@@ -66,6 +66,11 @@ update_missing|apply_or_skip),
# Test 'remote_apply' for 'insert_exists'
############################################
+# Change CONFLICT RESOLVER of insert_exists to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'remote_apply');"
+);
+
# Create local data on the subscriber
$node_subscriber->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
@@ -148,6 +153,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -202,16 +235,49 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#########################################
-# Test 'remote_apply' for 'delete_differ'
-#########################################
+#############################################
+# Test 'last_update_wins' for 'delete_differ'
+#############################################
+
+# Change CONFLICT RESOLVER of delete_differ to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'remote_apply');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -299,9 +365,9 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#########################################
-# Test 'remote_apply' for 'update_differ'
-#########################################
+#############################################
+# Test 'last_update_wins' for 'update_differ'
+#############################################
# Insert data in the publisher
$node_publisher->safe_psql(
@@ -329,6 +395,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'remote_apply');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
#########################################
# Test 'keep_local' for 'update_differ'
#########################################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2098ed7467..d42f32dab4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1565,6 +1565,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Please find v7 patch-set, the changes are:
Thanks Ajin for working on this. Please find few comments:
1)
parse_subscription_conflict_resolvers():
Here we loop in this function to find the given conflict type in the
supported list and error out if conflict-type is not valid. Also we
call validate_conflict_type_and_resolver() which again validates
conflict-type. I would recommend to loop 'stmtresolvers' in parse
function and then read each type and resolver and pass that to
validate_conflict_type_and_resolver(). Avoid double validation.
2)
SetSubConflictResolver():
It works well, but it does not look apt that the 'resolvers' passed to
this function by the caller is an array and this function knows the
array range and traverse from CT_MIN to CT_MAX assuming this array
maps directly to ConflictType. I think it would be better to have it
passed as a list and then SetSubConflictResolver() traverse the list
without knowing the range of it. Similar to what we do in
alter-sub-flow in and around UpdateSubConflictResolvers().
3)
When I execute 'alter subscription ..(detect_conflict=on)' for a
subscription which *already* has detect_conflict as ON, it tries to
reset resolvers to default and ends up in error. It should actually be
no-op in this particular situation and should not reset resolvers to
default.
postgres=# alter subscription sub1 set (detect_conflict=on);
WARNING: Using default conflict resolvers
ERROR: duplicate key value violates unique constraint
"pg_subscription_conflict_sub_index"
4)
Do we need SUBSCRIPTIONCONFLICTOID cache? We are not using it
anywhere. Shall we remove this and the corresponding index?
5)
RemoveSubscriptionConflictBySubid().
--We can remove extra blank line before table_open.
--We can get rid of curly braces around CatalogTupleDelete() as it is
a single line in loop.
thanks
Shveta
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Comment in 0002,
1) I do not see any test case that set a proper conflict type and
conflict resolver, all tests either give incorrect conflict
type/conflict resolver or the conflict resolver is ignored
0003
2) I was trying to think about this patch, so suppose we consider this
case conflict_type-> update_differ resolver->remote_apply, my
question is to confirm whether my understanding is correct. So if
this is set and we have 2 nodes and set up a 2-way logical
replication, and if a conflict occurs node-1 will take the changes of
node-2 and node-2 will take the changes of node-1? Maybe so I think
to avoid such cases user needs to set the resolver more thoughtfully,
on node-1 it may be set as "skip" and on node-1 as "remote-apply" so
in such cases if conflict happens both nodes will have the value from
node-1. But maybe it would be more difficult to get a consistent
value if we are setting up a mess replication topology right? Maybe
there I think a more advanced timestamp-based option would work better
IMHO.
I am doing code level review as well and will share my comments soon
on 0003 and 0004
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jul 30, 2024 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Comment in 0002,
1) I do not see any test case that set a proper conflict type and
conflict resolver, all tests either give incorrect conflict
type/conflict resolver or the conflict resolver is ignored0003
2) I was trying to think about this patch, so suppose we consider this
case conflict_type-> update_differ resolver->remote_apply, my
question is to confirm whether my understanding is correct. So if
this is set and we have 2 nodes and set up a 2-way logical
replication, and if a conflict occurs node-1 will take the changes of
node-2 and node-2 will take the changes of node-1?
Yes, that's right.
Maybe so I think
to avoid such cases user needs to set the resolver more thoughtfully,
on node-1 it may be set as "skip" and on node-1 as "remote-apply" so
in such cases if conflict happens both nodes will have the value from
node-1. But maybe it would be more difficult to get a consistent
value if we are setting up a mess replication topology right? Maybe
there I think a more advanced timestamp-based option would work better
IMHO.
Yes, that's correct. We can get data divergence with resolvers like
'remote_apply', 'keep_local' etc. If you meant 'mesh' replication
topology, then yes, it is difficult to get consistent value there with
resolvers other than timestamp based. And thus timestamp based
resolvers are needed and should be the default when implemented.
thanks
Shveta
On Tue, Jul 30, 2024 at 4:56 PM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Jul 30, 2024 at 4:04 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Comment in 0002,
1) I do not see any test case that set a proper conflict type and
conflict resolver, all tests either give incorrect conflict
type/conflict resolver or the conflict resolver is ignored0003
2) I was trying to think about this patch, so suppose we consider this
case conflict_type-> update_differ resolver->remote_apply, my
question is to confirm whether my understanding is correct. So if
this is set and we have 2 nodes and set up a 2-way logical
replication, and if a conflict occurs node-1 will take the changes of
node-2 and node-2 will take the changes of node-1?Yes, that's right.
Maybe so I think
to avoid such cases user needs to set the resolver more thoughtfully,
on node-1 it may be set as "skip" and on node-1 as "remote-apply" so
in such cases if conflict happens both nodes will have the value from
node-1. But maybe it would be more difficult to get a consistent
value if we are setting up a mess replication topology right? Maybe
there I think a more advanced timestamp-based option would work better
IMHO.Yes, that's correct. We can get data divergence with resolvers like
'remote_apply', 'keep_local' etc. If you meant 'mesh' replication
topology, then yes, it is difficult to get consistent value there with
resolvers other than timestamp based. And thus timestamp based
resolvers are needed and should be the default when implemented.
Thanks for the clarification.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
On Tue, Jul 30, 2024 at 2:19 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Please find v7 patch-set, the changes are:
Thanks Ajin for working on this. Please find few comments:
1)
parse_subscription_conflict_resolvers():
Here we loop in this function to find the given conflict type in the
supported list and error out if conflict-type is not valid. Also we
call validate_conflict_type_and_resolver() which again validates
conflict-type. I would recommend to loop 'stmtresolvers' in parse
function and then read each type and resolver and pass that to
validate_conflict_type_and_resolver(). Avoid double validation.
I have modified this as per comment.
2)
SetSubConflictResolver():
It works well, but it does not look apt that the 'resolvers' passed to
this function by the caller is an array and this function knows the
array range and traverse from CT_MIN to CT_MAX assuming this array
maps directly to ConflictType. I think it would be better to have it
passed as a list and then SetSubConflictResolver() traverse the list
without knowing the range of it. Similar to what we do in
alter-sub-flow in and around UpdateSubConflictResolvers().
I have kept the array as it requires that all conflict resolvers be set, if
not provided by the user then default needs to be used. However, I have
modified SetSubConflictResolver such that it takes in the size of the array
and does not assume it.
3)
When I execute 'alter subscription ..(detect_conflict=on)' for a
subscription which *already* has detect_conflict as ON, it tries to
reset resolvers to default and ends up in error. It should actually be
no-op in this particular situation and should not reset resolvers to
default.postgres=# alter subscription sub1 set (detect_conflict=on);
WARNING: Using default conflict resolvers
ERROR: duplicate key value violates unique constraint
"pg_subscription_conflict_sub_index"
fixed
4)
Do we need SUBSCRIPTIONCONFLICTOID cache? We are not using it
anywhere. Shall we remove this and the corresponding index?
We are using the index but not the cache, so removing the cache.
5)
RemoveSubscriptionConflictBySubid().
--We can remove extra blank line before table_open.
--We can get rid of curly braces around CatalogTupleDelete() as it is
a single line in loop.
fixed.
On Tue, Jul 30, 2024 at 8:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
On Fri, Jul 26, 2024 at 9:50 AM Ajin Cherian <itsajin@gmail.com> wrote:
Comment in 0002,
1) I do not see any test case that set a proper conflict type and
conflict resolver, all tests either give incorrect conflict
type/conflict resolver or the conflict resolver is ignored
fixed.
I've also fixed a cfbot error due to patch 0001. Rebase of table resolver
patch is still pending, will try and target that in the next patch-set.
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v8-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v8-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From cb9f46957122238076a28713ecfde1844e639e9c Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Wed, 31 Jul 2024 03:27:18 -0400
Subject: [PATCH v8 4/4] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
This patch also implements last_update_wins resolver.
Since conflict resolution for two phase commit transactions
using prepare-timestamp can result in data divergence, this patch
also restricts enabling two_phase and detect_conflict together
for a subscription.
---
src/backend/commands/subscriptioncmds.c | 45 ++++++
src/backend/executor/execReplication.c | 2 +-
.../replication/logical/applyparallelworker.c | 26 +++-
src/backend/replication/logical/conflict.c | 146 ++++++++++++++-----
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 155 +++++++++++++++++++--
src/backend/utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 ++++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/conflict.h | 6 +-
src/include/replication/logicalworker.h | 18 +++
src/include/replication/origin.h | 1 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
src/test/regress/expected/subscription.out | 46 +++---
src/test/subscription/t/029_on_error.pl | 53 +++++--
src/test/subscription/t/034_conflict_resolver.pl | 116 +++++++++++++--
src/tools/pgindent/typedefs.list | 1 +
18 files changed, 576 insertions(+), 93 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5c18c45..5a0c8b0 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -452,6 +453,22 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
"slot_name = NONE", "create_slot = false")));
}
}
+
+ /*
+ * Time based conflict resolution for two phase transactions can result in
+ * data divergence, so disallow enabling both together.
+ */
+ if (opts->detectconflict &&
+ IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ if (opts->twophase &&
+ IsSet(opts->specified_opts, SUBOPT_TWOPHASE_COMMIT))
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ /*- translator: both %s are strings of the form "option = value" */
+ errmsg("%s and %s are mutually exclusive options",
+ "detect_conflict = true", "two_phase = true")));
+ }
}
/*
@@ -719,6 +736,13 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
elog(WARNING, "subscriptions created by regression test cases should have names starting with \"regress_\"");
#endif
+ /* Warn if detect_conflict is enabled and track_commit_timestamp is off */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
+
rel = table_open(SubscriptionRelationId, RowExclusiveLock);
/* Check if name is used */
@@ -1454,6 +1478,27 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+
+ /*
+ * Time based conflict resolution for two phase
+ * transactions can result in data divergence, so disallow
+ * enabling it when two_phase is enabled.
+ */
+ if (sub->twophasestate == LOGICALREP_TWOPHASE_STATE_ENABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s for a subscription that has two_phase enabled",
+ "detect_conflict")));
+
+ /*
+ * Warn if detect_conflict is enabled and
+ * track_commit_timestamp is off.
+ */
+ if (opts.detectconflict && !track_commit_timestamp)
+ ereport(WARNING,
+ (errmsg("detect_conflict is enabled but \"%s\" is OFF, the last_update_wins resolution may not work",
+ "track_commit_timestamp"),
+ errhint("Enable \"%s\".", "track_commit_timestamp")));
}
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index fb96bb7..93ba364 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -669,7 +669,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool apply_remote = false;
GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ resolver = GetConflictResolver(*conflictslot, rel, CT_INSERT_EXISTS,
&apply_remote, NULL, subid);
ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c..10c7ca9 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -312,6 +312,20 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Don't start a new parallel worker if user has either configured max
+ * clock skew or if conflict detection and resolution is ON. In both cases
+ * we need commit timestamp in the beginning.
+ *
+ * XXX: For conflict reolution case, see if we can reduce the scope of
+ * this restriction to only such cases where time-based resolvers are
+ * actually being used.
+ */
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ MySubscription->detectconflict)
+ return false;
+
+
return true;
}
@@ -696,9 +710,19 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured, thus it is okay to pass 0 as origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with conflict
+ * detection enabled, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 9ac707c..7af07b8 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -50,6 +50,7 @@ const char *const ConflictTypeNames[] = {
const char *const ConflictResolverNames[] = {
[CR_REMOTE_APPLY] = "remote_apply",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -75,24 +76,24 @@ const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
* Default conflict resolver for each conflict type.
*/
const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_EXISTS] = CR_REMOTE_APPLY,
- [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_INSERT_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_EXISTS] = CR_LAST_UPDATE_WINS,
+ [CT_UPDATE_DIFFER] = CR_LAST_UPDATE_WINS,
[CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
[CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+ [CT_DELETE_DIFFER] = CR_LAST_UPDATE_WINS
};
@@ -207,6 +208,12 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
TupleTableSlot *conflictslot, bool apply_remote)
{
char *applymsg;
+ int errdet = 0;
+ char *local_ts;
+ char *remote_ts;
+
+ local_ts = pstrdup(timestamptz_to_str(localts));
+ remote_ts = pstrdup(timestamptz_to_str(replorigin_session_origin_timestamp));
if (apply_remote)
applymsg = "applying the remote changes.";
@@ -229,43 +236,63 @@ errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
if (resolver == CR_ERROR)
{
if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx),
+ localorigin, localxmin, local_ts);
else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ errdet = errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ errdet = errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
}
+ else if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Key already exists, %s. The local tuple : origin=%u, timestamp=%s; The remote tuple : origin=%u, timestamp=%s.",
+ applymsg, localorigin, local_ts,
+ replorigin_session_origin, remote_ts);
else
- return errdetail("Key already exists, %s", applymsg);
+ errdet = errdetail("Key already exists, %s", applymsg);
}
+ break;
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
case CT_UPDATE_MISSING:
if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
- return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ errdet = errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
else if (apply_remote)
- return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
else
- return errdetail("Did not find the row to be updated, %s",
- applymsg);
+ errdet = errdetail("Did not find the row to be updated, %s",
+ applymsg);
+ break;
case CT_DELETE_MISSING:
- return errdetail("Did not find the row to be deleted.");
+ errdet = errdetail("Did not find the row to be deleted.");
+ break;
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
- localorigin, localxmin,
- timestamptz_to_str(localts), applymsg);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s The remote tuple : origin=%u, timestamp=%s.",
+ localorigin, localxmin, local_ts, applymsg,
+ replorigin_session_origin, remote_ts);
+ else
+ errdet = errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin, local_ts, applymsg);
+ break;
}
- return 0; /* silence compiler warning */
+ pfree(local_ts);
+ pfree(remote_ts);
+
+ return errdet;
}
/*
@@ -375,6 +402,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -420,6 +456,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
}
/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleCommitTs(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
+/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
*/
@@ -610,7 +682,8 @@ RemoveSubscriptionConflictBySubid(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
@@ -619,6 +692,17 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_REMOTE_APPLY:
*apply_remote = true;
break;
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e481..bd8e6f0 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 36172b2..5b3b89e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,6 +319,20 @@ static uint32 parallel_stream_nchanges = 0;
bool InitializingApplyWorker = false;
/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
+/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
* Once we start skipping changes, we don't stop it until we skip all changes of
@@ -985,6 +999,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
}
/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
+/*
* Handle BEGIN message.
*/
static void
@@ -1005,6 +1108,15 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -1062,6 +1174,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1302,7 +1417,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -1999,7 +2115,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2053,6 +2170,16 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
/*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /* Capture the timestamp (prepare or commit) of the remote transaction */
+ replorigin_session_origin_timestamp = origin_timestamp;
+
+ /*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
*/
@@ -2157,7 +2284,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -2715,7 +2843,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -2755,7 +2883,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MyLogicalRepWorker->subid);
@@ -2909,7 +3037,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -2936,7 +3064,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -3135,7 +3263,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
*/
if (MySubscription->detectconflict)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MyLogicalRepWorker->subid);
@@ -3168,7 +3296,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_DIFFER,
&apply_remote, NULL,
MyLogicalRepWorker->subid);
@@ -4671,6 +4799,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4711,10 +4840,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index db37bee..a51f821 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 6a623f5..c9fdfa1 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3651,6 +3653,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4918,6 +4947,17 @@ struct config_enum ConfigureNamesEnum[] =
},
{
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
+ {
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 9ec9f97..f7a664a 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 2793ad6..8791f1b 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -59,6 +59,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -98,7 +101,8 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d..7cb0306 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9..dcbbbdf 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261..95b2a52 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03e..53b828d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 7bd4e82..f46e922 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -431,6 +431,8 @@ ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
WARNING: Will use default resolvers configuration as detect_conflict is ON but resolvers are not given
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
@@ -442,13 +444,13 @@ HINT: To initiate replication, you must manually create the replication slot, e
-- confirm that the default conflict resolvers have been set
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+---------------
- delete_differ | remote_apply
+ confrtype | confrres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
- update_exists | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | apply_or_skip
(6 rows)
@@ -487,18 +489,20 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--try setting resolvers for few types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
--check if above are configured; for non specified conflict types, default resolvers should be seen
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+--------------
+ confrtype | confrres
+----------------+------------------
delete_differ | keep_local
delete_missing | skip
insert_exists | keep_local
- update_differ | remote_apply
- update_exists | remote_apply
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | skip
(6 rows)
@@ -518,14 +522,16 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
-- setting detect_conflict to true will set default conflict resolvers
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
WARNING: Using default conflict resolvers
+WARNING: detect_conflict is enabled but "track_commit_timestamp" is OFF, the last_update_wins resolution may not work
+HINT: Enable "track_commit_timestamp".
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+---------------
- delete_differ | remote_apply
+ confrtype | confrres
+----------------+------------------
+ delete_differ | last_update_wins
delete_missing | skip
- insert_exists | remote_apply
- update_differ | remote_apply
- update_exists | remote_apply
+ insert_exists | last_update_wins
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | apply_or_skip
(6 rows)
@@ -538,13 +544,13 @@ ERROR: foo is not a valid conflict resolver
-- ok - valid conflict type and resolver
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
- confrtype | confrres
-----------------+--------------
+ confrtype | confrres
+----------------+------------------
delete_differ | keep_local
delete_missing | skip
insert_exists | keep_local
- update_differ | remote_apply
- update_exists | remote_apply
+ update_differ | last_update_wins
+ update_exists | last_update_wins
update_missing | skip
(6 rows)
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index e6f07fa..9620a4b 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -18,7 +18,8 @@ my $offset = 0;
# on the publisher.
sub test_skip_lsn
{
- my ($node_publisher, $node_subscriber, $nonconflict_data, $expected, $msg)
+ my ($node_publisher, $node_subscriber, $nonconflict_data, $expected,
+ $msg, $conflict_detection)
= @_;
# Wait until a conflict occurs on the subscriber.
@@ -26,13 +27,25 @@ sub test_skip_lsn
"SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'"
);
+ my $lsn;
+ my $contents = slurp_file($node_subscriber->logfile, $offset);
+
# Get the finish LSN of the error transaction, mapping the expected
# ERROR with its CONTEXT when retrieving this information.
- my $contents = slurp_file($node_subscriber->logfile, $offset);
- $contents =~
- qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
- or die "could not get error-LSN";
- my $lsn = $1;
+ if ($conflict_detection)
+ {
+ $contents =~
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
+ else
+ {
+ $contents =~
+ qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ or die "could not get error-LSN";
+ $lsn = $1;
+ }
# Set skip lsn.
$node_subscriber->safe_psql('postgres',
@@ -110,7 +123,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, detect_conflict = on)"
);
# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
@@ -148,7 +161,22 @@ INSERT INTO tbl VALUES (1, NULL);
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(2, NULL)", "2", "test skipping transaction");
+ "(2, NULL)", "2", "test skipping transaction", 1);
+
+# Cleanup before we start PREPARE AND COMMIT PREPARED tests
+$node_subscriber->safe_psql('postgres', "TRUNCATE tbl");
+$node_publisher->safe_psql('postgres', "TRUNCATE tbl");
+
+# Drop subscription and recreate with two_phase enabled
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub");
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+);
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, 'sub');
+
+$node_subscriber->safe_psql('postgres', "INSERT INTO tbl VALUES (1, NULL)");
# Test for PREPARE and COMMIT PREPARED. Insert the same data to tbl and
# PREPARE the transaction, raising an error. Then skip the transaction.
@@ -161,7 +189,7 @@ PREPARE TRANSACTION 'gtx';
COMMIT PREPARED 'gtx';
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(3, NULL)", "3", "test skipping prepare and commit prepared ");
+ "(2, NULL)", "2", "test skipping prepare and commit prepared ", 0);
# Test for STREAM COMMIT. Insert enough rows to tbl to exceed the 64kB
# limit, also raising an error on the subscriber during applying spooled
@@ -174,17 +202,14 @@ INSERT INTO tbl SELECT i, sha256(i::text::bytea) FROM generate_series(1, 10000)
COMMIT;
]);
test_skip_lsn($node_publisher, $node_subscriber,
- "(4, sha256(4::text::bytea))",
- "4", "test skipping stream-commit");
+ "(3, sha256(4::text::bytea))",
+ "3", "test skipping stream-commit", 0);
$result = $node_subscriber->safe_psql('postgres',
"SELECT COUNT(*) FROM pg_prepared_xacts");
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
-# Reset conflict resolver for 'insert_exist' conflict type to default.
-$node_subscriber->safe_psql('postgres',
- "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=remote_apply)");
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 58751c5..e961809 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -54,11 +54,11 @@ $node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
my $result = $node_subscriber->safe_psql('postgres',
"SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
);
-is( $result, qq(delete_differ|remote_apply
+is( $result, qq(delete_differ|last_update_wins
delete_missing|skip
-insert_exists|remote_apply
-update_differ|remote_apply
-update_exists|remote_apply
+insert_exists|last_update_wins
+update_differ|last_update_wins
+update_exists|last_update_wins
update_missing|apply_or_skip),
"confirm that the default conflict resolvers are in place");
@@ -66,6 +66,11 @@ update_missing|apply_or_skip),
# Test 'remote_apply' for 'insert_exists'
############################################
+# Change CONFLICT RESOLVER of insert_exists to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'remote_apply');"
+);
+
# Create local data on the subscriber
$node_subscriber->safe_psql('postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
@@ -148,6 +153,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -202,16 +235,49 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#########################################
-# Test 'remote_apply' for 'delete_differ'
-#########################################
+#############################################
+# Test 'last_update_wins' for 'delete_differ'
+#############################################
+
+# Change CONFLICT RESOLVER of delete_differ to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'remote_apply');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -299,9 +365,9 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#########################################
-# Test 'remote_apply' for 'update_differ'
-#########################################
+#############################################
+# Test 'last_update_wins' for 'update_differ'
+#############################################
# Insert data in the publisher
$node_publisher->safe_psql(
@@ -330,6 +396,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to remote_apply
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'remote_apply');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
+#########################################
# Test 'keep_local' for 'update_differ'
#########################################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ae1db1a..77c6469 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1565,6 +1565,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
1.8.3.1
v8-0002-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchapplication/octet-stream; name=v8-0002-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchDownload
From d5cabc6ea1b2c680efcf6012ec89b6da5525c8ea Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Wed, 31 Jul 2024 03:08:50 -0400
Subject: [PATCH v8 2/4] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
---
src/backend/commands/subscriptioncmds.c | 100 ++++++++
src/backend/parser/gram.y | 25 +-
src/backend/replication/logical/conflict.c | 308 +++++++++++++++++++++++++
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_subscription_conflict.h | 55 +++++
src/include/nodes/parsenodes.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 ++++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 101 ++++++++
src/test/regress/sql/subscription.sql | 48 ++++
12 files changed, 691 insertions(+), 4 deletions(-)
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b2bc095..5c18c45 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -28,6 +28,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +38,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -453,6 +455,32 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+
+ if (!stmtresolvers)
+ return;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+
+ /* validate the conflict type and resolver */
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* update the corresponding resolver for the given conflict type */
+ resolvers[type].resolver = defGetString(defel);
+ }
+}
+
+/*
* Add publication names from the list to a string.
*/
static void
@@ -596,6 +624,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+ bool skip_conflict_resolvers = false;
/*
* Parse and check options.
@@ -611,6 +641,33 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ if (!opts.detectconflict)
+ {
+ /*
+ * If conflict resolvers are set but detect_conflict is not enabled,
+ * issue warning and ignore resolvers.
+ */
+ if (stmt->resolvers)
+ ereport(WARNING,
+ (errmsg("Ignoring given CONFLICT RESOLVERS as detect_conflict is not enabled.")));
+ skip_conflict_resolvers = true;
+ }
+ else
+ {
+ /* If conflict resolvers are not set, use default values */
+ if (!stmt->resolvers)
+ {
+ ereport(WARNING,
+ (errmsg("Will use default resolvers configuration as detect_conflict is ON but resolvers are not given")));
+ }
+
+ /*
+ * Parse and check conflict resolvers. Initialize with default values
+ */
+ SetDefaultResolvers(conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ }
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -742,6 +799,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ if (!skip_conflict_resolvers)
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
/*
* Connect to remote side to execute requested commands and fetch table
* info.
@@ -1374,6 +1435,22 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
{
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+
+ if (!opts.detectconflict)
+ RemoveSubscriptionConflictBySubid(subid);
+ else
+ {
+ /* if detect_conflict is already set, then ignore */
+ if (!sub->detectconflict)
+ {
+ ereport(WARNING,
+ (errmsg("Using default conflict resolvers")));
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+ }
+ }
+
values[Anum_pg_subscription_subdetectconflict - 1] =
BoolGetDatum(opts.detectconflict);
replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
@@ -1604,6 +1681,26 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /* make sure that detect_conflict is enabled, else throw error */
+ if (!sub->detectconflict)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set conflict resolvers when detect_conflict is not enabled")));
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1855,6 +1952,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a043fd4..a690b46 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,7 +426,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -772,7 +772,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECHECK RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8824,6 +8824,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10758,7 +10763,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10766,6 +10771,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10872,6 +10878,17 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+
;
/*****************************************************************************
@@ -17797,6 +17814,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18428,6 +18446,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 4918011..720e923 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,9 +15,23 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "replication/conflict.h"
#include "replication/origin.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
#include "utils/rel.h"
const char *const ConflictTypeNames[] = {
@@ -29,6 +43,55 @@ const char *const ConflictTypeNames[] = {
[CT_DELETE_DIFFER] = "delete_differ"
};
+const char *const ConflictResolverNames[] = {
+ [CR_REMOTE_APPLY] = "remote_apply",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain remote_apply and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * remote_apply: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_REMOTE_APPLY, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_EXISTS] = CR_REMOTE_APPLY,
+ [CT_UPDATE_DIFFER] = CR_REMOTE_APPLY,
+ [CT_UPDATE_MISSING] = CR_APPLY_OR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_REMOTE_APPLY
+
+};
+
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
TransactionId localxmin,
@@ -191,3 +254,248 @@ build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
return conflict_row;
}
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *CTR = NULL;
+ List *res = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ CTR = palloc(sizeof(ConflictTypeResolver));
+ CTR->conflict_type = defel->defname;
+ CTR->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+
+ res = lappend(res, CTR);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolver for a conflict type in
+ * pg_subscription_conflict system catalog
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int type;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ for (type = 0; type < resolvers_cnt; type++)
+ {
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[type].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[type].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a..f2611c1 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1da..959e1d9 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000..c8b37c2
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTSUBOID, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b5..2e1eefc3 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4203,6 +4203,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4215,6 +4216,7 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4225,6 +4227,7 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f7fe834..21e5cb1 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -377,6 +377,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 3a7260d..c102f74 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -39,6 +39,47 @@ typedef enum
CT_DELETE_DIFFER,
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_DIFFER
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_REMOTE_APPLY = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_REMOTE_APPLY
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
extern void ReportApplyConflict(int elevel, ConflictType type,
@@ -46,5 +87,13 @@ extern void ReportApplyConflict(int elevel, ConflictType type,
TransactionId localxmin, RepOriginId localorigin,
TimestampTz localts, TupleTableSlot *conflictslot);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictBySubid(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb89..42bf2cc 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index cc9337c..7bd4e82 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -430,6 +430,7 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ERROR: detect_conflict requires a Boolean value
-- now it works
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: Will use default resolvers configuration as detect_conflict is ON but resolvers are not given
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
@@ -439,6 +440,18 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
(1 row)
+-- confirm that the default conflict resolvers have been set
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | apply_or_skip
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
\dRs+
List of subscriptions
@@ -447,6 +460,94 @@ ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
+-- confirm that the conflict resolvers have been dropped
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = foo);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- creating subscription with detect_conflict = false should not create conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | keep_local
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | skip
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- creating subscription with detect_conflict = false but conflict resolvers specified
+-- will result in conflict resolvers being ignored
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false) CONFLICT RESOLVER (insert_exists = 'keep_local');
+WARNING: Ignoring given CONFLICT RESOLVERS as detect_conflict is not enabled.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------+----------
+(0 rows)
+
+-- setting detect_conflict to true will set default conflict resolvers
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+WARNING: Using default conflict resolvers
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+---------------
+ delete_differ | remote_apply
+ delete_missing | skip
+ insert_exists | remote_apply
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | apply_or_skip
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | keep_local
+ update_differ | remote_apply
+ update_exists | remote_apply
+ update_missing | skip
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- let's do some tests with pg_create_subscription rather than superuser
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 5c740fd..bdd2118 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -295,10 +295,58 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
\dRs+
+-- confirm that the default conflict resolvers have been set
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
\dRs+
+-- confirm that the conflict resolvers have been dropped
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- creating subscription with detect_conflict = false should not create conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- creating subscription with detect_conflict = false but conflict resolvers specified
+-- will result in conflict resolvers being ignored
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = false) CONFLICT RESOLVER (insert_exists = 'keep_local');
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- setting detect_conflict to true will set default conflict resolvers
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = true);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--
1.8.3.1
v8-0001-Detect-and-log-conflicts-in-logical-replication.patchapplication/octet-stream; name=v8-0001-Detect-and-log-conflicts-in-logical-replication.patchDownload
From 65e60fa35df180ccf781363d6c2d363c0be36637 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Wed, 31 Jul 2024 03:01:17 -0400
Subject: [PATCH v8 1/4] Detect and log conflicts in logical replication
This patch adds a new parameter detect_conflict for CREATE and ALTER
subscription commands. This new parameter will decide if subscription will
go for confict detection. By default, conflict detection will be off for a
subscription.
When conflict detection is enabled, additional logging is triggered in the
following conflict scenarios:
insert_exists: Inserting a row that violates a NOT DEFERRABLE unique constraint.
update_exists: The updated row value violates a NOT DEFERRABLE unique constraint.
update_differ: Updating a row that was previously modified by another origin.
update_missing: The tuple to be updated is missing.
delete_missing: The tuple to be deleted is missing.
delete_differ: Deleting a row that was previously modified by another origin.
For insert_exists conflict, the log can include origin and commit
timestamp details of the conflicting key with track_commit_timestamp
enabled.
update_differ and delete_differ conflicts can only be detected when
track_commit_timestamp is enabled.
---
doc/src/sgml/catalogs.sgml | 9 ++
doc/src/sgml/logical-replication.sgml | 21 ++-
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 84 ++++++++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 3 +-
src/backend/commands/subscriptioncmds.c | 31 ++++-
src/backend/executor/execIndexing.c | 14 +-
src/backend/executor/execReplication.c | 142 +++++++++++++++++++-
src/backend/replication/logical/Makefile | 1 +
src/backend/replication/logical/conflict.c | 193 ++++++++++++++++++++++++++++
src/backend/replication/logical/meson.build | 1 +
src/backend/replication/logical/worker.c | 84 +++++++++---
src/bin/pg_dump/pg_dump.c | 17 ++-
src/bin/pg_dump/pg_dump.h | 1 +
src/bin/psql/describe.c | 6 +-
src/bin/psql/tab-complete.c | 14 +-
src/include/catalog/pg_subscription.h | 4 +
src/include/replication/conflict.h | 50 +++++++
src/test/regress/expected/subscription.out | 176 ++++++++++++++-----------
src/test/regress/sql/subscription.sql | 15 +++
src/test/subscription/t/001_rep_changes.pl | 15 ++-
src/test/subscription/t/013_partition.pl | 68 ++++++----
src/test/subscription/t/029_on_error.pl | 5 +-
src/test/subscription/t/030_origin.pl | 43 +++++++
src/tools/pgindent/typedefs.list | 1 +
26 files changed, 849 insertions(+), 155 deletions(-)
create mode 100644 src/backend/replication/logical/conflict.c
create mode 100644 src/include/replication/conflict.h
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index b654fae..b042a5a 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -8037,6 +8037,15 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<row>
<entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subdetectconflict</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription is enabled for conflict detection.
+ </para></entry>
+ </row>
+
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
</para>
<para>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index a23a3d5..d236d85 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1580,7 +1580,8 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
stop. This is referred to as a <firstterm>conflict</firstterm>. When
replicating <command>UPDATE</command> or <command>DELETE</command>
operations, missing data will not produce a conflict and such operations
- will simply be skipped.
+ will simply be skipped. Please refer to <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ for all the conflicts that will be logged when enabling <literal>detect_conflict</literal>.
</para>
<para>
@@ -1649,6 +1650,24 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
linkend="sql-altersubscription"><command>ALTER SUBSCRIPTION ...
SKIP</command></link>.
</para>
+
+ <para>
+ Enabling both <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>
+ and <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ on the subscriber can provide additional details regarding conflicting
+ rows, such as their origin and commit timestamp, in case of a unique
+ constraint violation conflict:
+<screen>
+ERROR: conflict insert_exists detected on relation "public.t"
+DETAIL: Key (a)=(1) already exists in unique index "t_pkey", which was modified by origin 0 in transaction 740 at 2024-06-26 10:47:04.727375+08.
+CONTEXT: processing remote data for replication origin "pg_16389" during message type "INSERT" for replication target relation "public.t" in transaction 740, finished at 0/14F7EC0
+</screen>
+ Users can use these information to make decisions on whether to retain
+ the local change or adopt the remote alteration. For instance, the
+ origin in above log indicates that the existing row was modified by a
+ local change, users can manually perform a remote-change-win resolution
+ by deleting the local row.
+ </para>
</sect1>
<sect1 id="logical-replication-restrictions">
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d..dfbe25b 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -235,8 +235,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<link linkend="sql-createsubscription-params-with-password-required"><literal>password_required</literal></link>,
<link linkend="sql-createsubscription-params-with-run-as-owner"><literal>run_as_owner</literal></link>,
<link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link>,
- <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>, and
- <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>.
+ <link linkend="sql-createsubscription-params-with-failover"><literal>failover</literal></link>,
+ <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, and
+ <link linkend="sql-createsubscription-params-with-detect-conflict"><literal>detect_conflict</literal></link>.
Only a superuser can set <literal>password_required = false</literal>.
</para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 740b7d94..eb51805 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -428,6 +428,90 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-detect-conflict">
+ <term><literal>detect_conflict</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription is enabled for conflict detection.
+ The default is <literal>false</literal>.
+ </para>
+ <para>
+ When conflict detection is enabled, additional logging is triggered
+ in the following scenarios:
+ <variablelist>
+ <varlistentry>
+ <term><literal>insert_exists</literal></term>
+ <listitem>
+ <para>
+ Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_exists</literal></term>
+ <listitem>
+ <para>
+ The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint. Note that to obtain the origin and commit
+ timestamp details of the conflicting key in the log, ensure that
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. In this scenario, an error will be raised until the
+ conflict is resolved manually.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_differ</literal></term>
+ <listitem>
+ <para>
+ Updating a row that was previously modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the update is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>update_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be updated was not found. The update will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_missing</literal></term>
+ <listitem>
+ <para>
+ The tuple to be deleted was not found. The delete will simply be
+ skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>delete_differ</literal></term>
+ <listitem>
+ <para>
+ Deleting a row that was previously modified by another origin. Note that this
+ conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled. Currenly, the delete is always applied regardless of
+ the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist></para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 9efc915..5a423f4 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->passwordrequired = subform->subpasswordrequired;
sub->runasowner = subform->subrunasowner;
sub->failover = subform->subfailover;
+ sub->detectconflict = subform->subdetectconflict;
/* Get conninfo */
datum = SysCacheGetAttrNotNull(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 19cabc9..d084bfc 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1356,7 +1356,8 @@ REVOKE ALL ON pg_subscription FROM public;
GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled,
subbinary, substream, subtwophasestate, subdisableonerr,
subpasswordrequired, subrunasowner, subfailover,
- subslotname, subsynccommit, subpublications, suborigin)
+ subdetectconflict, subslotname, subsynccommit,
+ subpublications, suborigin)
ON pg_subscription TO public;
CREATE VIEW pg_stat_subscription_stats AS
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d124bfe..b2bc095 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -71,8 +71,9 @@
#define SUBOPT_PASSWORD_REQUIRED 0x00000800
#define SUBOPT_RUN_AS_OWNER 0x00001000
#define SUBOPT_FAILOVER 0x00002000
-#define SUBOPT_LSN 0x00004000
-#define SUBOPT_ORIGIN 0x00008000
+#define SUBOPT_DETECT_CONFLICT 0x00004000
+#define SUBOPT_LSN 0x00008000
+#define SUBOPT_ORIGIN 0x00010000
/* check if the 'val' has 'bits' set */
#define IsSet(val, bits) (((val) & (bits)) == (bits))
@@ -98,6 +99,7 @@ typedef struct SubOpts
bool passwordrequired;
bool runasowner;
bool failover;
+ bool detectconflict;
char *origin;
XLogRecPtr lsn;
} SubOpts;
@@ -162,6 +164,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->runasowner = false;
if (IsSet(supported_opts, SUBOPT_FAILOVER))
opts->failover = false;
+ if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT))
+ opts->detectconflict = false;
if (IsSet(supported_opts, SUBOPT_ORIGIN))
opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY);
@@ -307,6 +311,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
opts->specified_opts |= SUBOPT_FAILOVER;
opts->failover = defGetBoolean(defel);
}
+ else if (IsSet(supported_opts, SUBOPT_DETECT_CONFLICT) &&
+ strcmp(defel->defname, "detect_conflict") == 0)
+ {
+ if (IsSet(opts->specified_opts, SUBOPT_DETECT_CONFLICT))
+ errorConflictingDefElem(defel, pstate);
+
+ opts->specified_opts |= SUBOPT_DETECT_CONFLICT;
+ opts->detectconflict = defGetBoolean(defel);
+ }
else if (IsSet(supported_opts, SUBOPT_ORIGIN) &&
strcmp(defel->defname, "origin") == 0)
{
@@ -594,7 +607,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_SYNCHRONOUS_COMMIT | SUBOPT_BINARY |
SUBOPT_STREAMING | SUBOPT_TWOPHASE_COMMIT |
SUBOPT_DISABLE_ON_ERR | SUBOPT_PASSWORD_REQUIRED |
- SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
+ SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
@@ -701,6 +715,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
values[Anum_pg_subscription_subpasswordrequired - 1] = BoolGetDatum(opts.passwordrequired);
values[Anum_pg_subscription_subrunasowner - 1] = BoolGetDatum(opts.runasowner);
values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover);
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (opts.slot_name)
@@ -1196,7 +1212,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
SUBOPT_DISABLE_ON_ERR |
SUBOPT_PASSWORD_REQUIRED |
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER |
- SUBOPT_ORIGIN);
+ SUBOPT_DETECT_CONFLICT | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options,
supported_opts, &opts);
@@ -1356,6 +1372,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
replaces[Anum_pg_subscription_subfailover - 1] = true;
}
+ if (IsSet(opts.specified_opts, SUBOPT_DETECT_CONFLICT))
+ {
+ values[Anum_pg_subscription_subdetectconflict - 1] =
+ BoolGetDatum(opts.detectconflict);
+ replaces[Anum_pg_subscription_subdetectconflict - 1] = true;
+ }
+
if (IsSet(opts.specified_opts, SUBOPT_ORIGIN))
{
values[Anum_pg_subscription_suborigin - 1] =
diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c
index 9f05b36..ef52277 100644
--- a/src/backend/executor/execIndexing.c
+++ b/src/backend/executor/execIndexing.c
@@ -207,8 +207,9 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative)
ii = BuildIndexInfo(indexDesc);
/*
- * If the indexes are to be used for speculative insertion, add extra
- * information required by unique index entries.
+ * If the indexes are to be used for speculative insertion or conflict
+ * detection in logical replication, add extra information required by
+ * unique index entries.
*/
if (speculative && ii->ii_Unique)
BuildSpeculativeIndexInfo(indexDesc, ii);
@@ -521,6 +522,11 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo,
* possible that a conflicting tuple is inserted immediately
* after this returns. But this can be used for a pre-check
* before insertion.
+ *
+ * If the 'slot' holds a tuple with valid tid, this tuple will
+ * be ignored when checking conflict. This can help in scenarios
+ * where we want to re-check for conflicts after inserting a
+ * tuple.
* ----------------------------------------------------------------
*/
bool
@@ -536,11 +542,9 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
ExprContext *econtext;
Datum values[INDEX_MAX_KEYS];
bool isnull[INDEX_MAX_KEYS];
- ItemPointerData invalidItemPtr;
bool checkedIndex = false;
ItemPointerSetInvalid(conflictTid);
- ItemPointerSetInvalid(&invalidItemPtr);
/*
* Get information from the result relation info structure.
@@ -629,7 +633,7 @@ ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot,
satisfiesConstraint =
check_exclusion_or_unique_constraint(heapRelation, indexRelation,
- indexInfo, &invalidItemPtr,
+ indexInfo, &slot->tts_tid,
values, isnull, estate, false,
CEOUC_WAIT, true,
conflictTid);
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index d0a89cd..0680bc8 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -23,6 +23,7 @@
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
+#include "replication/conflict.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/builtins.h"
@@ -481,6 +482,121 @@ retry:
}
/*
+ * Find the tuple that violates the passed in unique index constraint
+ * (conflictindex).
+ *
+ * Return false if there is no conflict and *conflictslot is set to NULL.
+ * Otherwise return true, and the conflicting tuple is locked and returned
+ * in *conflictslot.
+ */
+static bool
+FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
+ Oid conflictindex, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot)
+{
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ ItemPointerData conflictTid;
+ TM_FailureData tmfd;
+ TM_Result res;
+
+ *conflictslot = NULL;
+
+retry:
+ if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
+ &conflictTid, list_make1_oid(conflictindex)))
+ {
+ if (*conflictslot)
+ ExecDropSingleTupleTableSlot(*conflictslot);
+
+ *conflictslot = NULL;
+ return false;
+ }
+
+ *conflictslot = table_slot_create(rel, NULL);
+
+ PushActiveSnapshot(GetLatestSnapshot());
+
+ res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
+ *conflictslot,
+ GetCurrentCommandId(false),
+ LockTupleShare,
+ LockWaitBlock,
+ 0 /* don't follow updates */ ,
+ &tmfd);
+
+ PopActiveSnapshot();
+
+ switch (res)
+ {
+ case TM_Ok:
+ break;
+ case TM_Updated:
+ /* XXX: Improve handling here */
+ if (ItemPointerIndicatesMovedPartitions(&tmfd.ctid))
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("tuple to be locked was already moved to another partition due to concurrent update, retrying")));
+ else
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent update, retrying")));
+ goto retry;
+ case TM_Deleted:
+ /* XXX: Improve handling here */
+ ereport(LOG,
+ (errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
+ errmsg("concurrent delete, retrying")));
+ goto retry;
+ case TM_Invisible:
+ elog(ERROR, "attempted to lock invisible tuple");
+ break;
+ default:
+ elog(ERROR, "unexpected table_tuple_lock status: %u", res);
+ break;
+ }
+
+ return true;
+}
+
+/*
+ * Re-check all the unique indexes in 'recheckIndexes' to see if there are
+ * potential conflicts with the tuple in 'slot'.
+ *
+ * This function is invoked after inserting or updating a tuple that detected
+ * potential conflict tuples. It attempts to find the tuple that conflicts with
+ * the provided tuple. This operation may seem redundant with the unique
+ * violation check of indexam, but since we call this function only when we are
+ * detecting conflict in logical replication and encountering potential
+ * conflicts with any unique index constraints (which should not be frequent),
+ * so it's ok. Moreover, upon detecting a conflict, we will report an ERROR and
+ * restart the logical replication, so the additional cost of finding the tuple
+ * should be acceptable.
+ */
+static void
+ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
+ ConflictType type, List *recheckIndexes,
+ TupleTableSlot *slot)
+{
+ /* Re-check all the unique indexes for potential conflicts */
+ foreach_oid(uniqueidx, resultRelInfo->ri_onConflictArbiterIndexes)
+ {
+ TupleTableSlot *conflictslot;
+
+ if (list_member_oid(recheckIndexes, uniqueidx) &&
+ FindConflictTuple(resultRelInfo, estate, uniqueidx, slot, &conflictslot))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
+ xmin, origin, committs, conflictslot);
+ }
+ }
+}
+
+/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
*
@@ -509,6 +625,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (!skip_tuple)
{
List *recheckIndexes = NIL;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -525,10 +643,17 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, false, false,
- NULL, NIL, false);
+ slot, estate, false,
+ conflictindexes, &conflict,
+ conflictindexes, false);
+
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_INSERT_EXISTS,
+ recheckIndexes, slot);
/* AFTER ROW INSERT Triggers */
ExecARInsertTriggers(estate, resultRelInfo, slot,
@@ -577,6 +702,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
{
List *recheckIndexes = NIL;
TU_UpdateIndexes update_indexes;
+ List *conflictindexes;
+ bool conflict = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -593,12 +720,19 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
- slot, estate, true, false,
- NULL, NIL,
+ slot, estate, true,
+ conflictindexes,
+ &conflict, conflictindexes,
(update_indexes == TU_Summarizing));
+ if (conflict)
+ ReCheckConflictIndexes(resultRelInfo, estate, CT_UPDATE_EXISTS,
+ recheckIndexes, slot);
+
/* AFTER ROW UPDATE Triggers */
ExecARUpdateTriggers(estate, resultRelInfo,
NULL, NULL,
diff --git a/src/backend/replication/logical/Makefile b/src/backend/replication/logical/Makefile
index ba03eef..1e08bbb 100644
--- a/src/backend/replication/logical/Makefile
+++ b/src/backend/replication/logical/Makefile
@@ -16,6 +16,7 @@ override CPPFLAGS := -I$(srcdir) $(CPPFLAGS)
OBJS = \
applyparallelworker.o \
+ conflict.o \
decode.o \
launcher.o \
logical.o \
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
new file mode 100644
index 0000000..4918011
--- /dev/null
+++ b/src/backend/replication/logical/conflict.c
@@ -0,0 +1,193 @@
+/*-------------------------------------------------------------------------
+ * conflict.c
+ * Functionality for detecting and logging conflicts.
+ *
+ * Copyright (c) 2024, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/replication/logical/conflict.c
+ *
+ * This file contains the code for detecting and logging conflicts on
+ * the subscriber during logical replication.
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "access/commit_ts.h"
+#include "replication/conflict.h"
+#include "replication/origin.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+
+const char *const ConflictTypeNames[] = {
+ [CT_INSERT_EXISTS] = "insert_exists",
+ [CT_UPDATE_EXISTS] = "update_exists",
+ [CT_UPDATE_DIFFER] = "update_differ",
+ [CT_UPDATE_MISSING] = "update_missing",
+ [CT_DELETE_MISSING] = "delete_missing",
+ [CT_DELETE_DIFFER] = "delete_differ"
+};
+
+static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
+static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin,
+ RepOriginId localorigin,
+ TimestampTz localts,
+ TupleTableSlot *conflictslot);
+
+/*
+ * Get the xmin and commit timestamp data (origin and timestamp) associated
+ * with the provided local tuple.
+ *
+ * Return true if the commit timestamp data was found, false otherwise.
+ */
+bool
+GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts)
+{
+ Datum xminDatum;
+ bool isnull;
+
+ xminDatum = slot_getsysattr(localslot, MinTransactionIdAttributeNumber,
+ &isnull);
+ *xmin = DatumGetTransactionId(xminDatum);
+ Assert(!isnull);
+
+ /*
+ * The commit timestamp data is not available if track_commit_timestamp is
+ * disabled.
+ */
+ if (!track_commit_timestamp)
+ {
+ *localorigin = InvalidRepOriginId;
+ *localts = 0;
+ return false;
+ }
+
+ return TransactionIdGetCommitTsData(*xmin, localts, localorigin);
+}
+
+/*
+ * Report a conflict when applying remote changes.
+ */
+void
+ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot)
+{
+ ereport(elevel,
+ errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
+ errmsg("conflict %s detected on relation \"%s.%s\"",
+ ConflictTypeNames[type],
+ get_namespace_name(RelationGetNamespace(localrel)),
+ RelationGetRelationName(localrel)),
+ errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
+ localts, conflictslot));
+}
+
+/*
+ * Find all unique indexes to check for a conflict and store them into
+ * ResultRelInfo.
+ */
+void
+InitConflictIndexes(ResultRelInfo *relInfo)
+{
+ List *uniqueIndexes = NIL;
+
+ for (int i = 0; i < relInfo->ri_NumIndices; i++)
+ {
+ Relation indexRelation = relInfo->ri_IndexRelationDescs[i];
+
+ if (indexRelation == NULL)
+ continue;
+
+ /* Detect conflict only for unique indexes */
+ if (!relInfo->ri_IndexRelationInfo[i]->ii_Unique)
+ continue;
+
+ /* Don't support conflict detection for deferrable index */
+ if (!indexRelation->rd_index->indimmediate)
+ continue;
+
+ uniqueIndexes = lappend_oid(uniqueIndexes,
+ RelationGetRelid(indexRelation));
+ }
+
+ relInfo->ri_onConflictArbiterIndexes = uniqueIndexes;
+}
+
+/*
+ * Add an errdetail() line showing conflict detail.
+ */
+static int
+errdetail_apply_conflict(ConflictType type, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot)
+{
+ switch (type)
+ {
+ case CT_INSERT_EXISTS:
+ case CT_UPDATE_EXISTS:
+ {
+ /*
+ * Bulid the index value string. If the return value is NULL,
+ * it indicates that the current user lacks permissions to
+ * view all the columns involved.
+ */
+ char *index_value = build_index_value_desc(conflictidx,
+ conflictslot);
+
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
+ case CT_UPDATE_DIFFER:
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ case CT_UPDATE_MISSING:
+ return errdetail("Did not find the row to be updated.");
+ case CT_DELETE_MISSING:
+ return errdetail("Did not find the row to be deleted.");
+ case CT_DELETE_DIFFER:
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
+ localorigin, localxmin, timestamptz_to_str(localts));
+ }
+
+ return 0; /* silence compiler warning */
+}
+
+/*
+ * Helper functions to construct a string describing the contents of an index
+ * entry. See BuildIndexValueDescription for details.
+ */
+static char *
+build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot)
+{
+ char *conflict_row;
+ Relation indexDesc;
+
+ if (!conflictslot)
+ return NULL;
+
+ /* Assume the index has been locked */
+ indexDesc = index_open(indexoid, NoLock);
+
+ slot_getallattrs(conflictslot);
+
+ conflict_row = BuildIndexValueDescription(indexDesc,
+ conflictslot->tts_values,
+ conflictslot->tts_isnull);
+
+ index_close(indexDesc, NoLock);
+
+ return conflict_row;
+}
diff --git a/src/backend/replication/logical/meson.build b/src/backend/replication/logical/meson.build
index 3dec36a..3d36249 100644
--- a/src/backend/replication/logical/meson.build
+++ b/src/backend/replication/logical/meson.build
@@ -2,6 +2,7 @@
backend_sources += files(
'applyparallelworker.c',
+ 'conflict.c',
'decode.c',
'launcher.c',
'logical.c',
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ec96b5f..fc3f80e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -167,6 +167,7 @@
#include "postmaster/bgworker.h"
#include "postmaster/interrupt.h"
#include "postmaster/walwriter.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -2458,7 +2459,10 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
EState *estate = edata->estate;
/* We must open indexes here. */
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
+
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
@@ -2646,7 +2650,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
MemoryContext oldctx;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(relinfo, false);
+ ExecOpenIndices(relinfo, MySubscription->detectconflict);
found = FindReplTupleInLocalRel(edata, localrel,
&relmapentry->remoterel,
@@ -2661,6 +2665,20 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2668,6 +2686,9 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualSetSlot(&epqstate, remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
@@ -2678,13 +2699,10 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2807,6 +2825,20 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/* If found delete it. */
if (found)
{
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
+
+ /*
+ * If conflict detection is enabled, check whether the local tuple was
+ * modified by a different origin. If detected, report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
+ localxmin, localorigin, localts, NULL);
+
EvalPlanQualSetSlot(&epqstate, localslot);
/* Do the actual delete. */
@@ -2818,13 +2850,10 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
/*
* The tuple to be deleted could not be found. Do nothing except for
* emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be deleted "
- "in replication target relation \"%s\"",
- RelationGetRelationName(localrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId, 0, NULL);
}
/* Cleanup. */
@@ -2991,6 +3020,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ResultRelInfo *partrelinfo_new;
Relation partrel_new;
bool found;
+ RepOriginId localorigin;
+ TransactionId localxmin;
+ TimestampTz localts;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3002,17 +3034,29 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/*
* The tuple to be updated could not be found. Do nothing
* except for emitting a log message.
- *
- * XXX should this be promoted to ereport(LOG) perhaps?
*/
- elog(DEBUG1,
- "logical replication did not find row to be updated "
- "in replication target relation's partition \"%s\"",
- RelationGetRelationName(partrel));
+ if (MySubscription->detectconflict)
+ ReportApplyConflict(LOG, CT_UPDATE_MISSING,
+ partrel, InvalidOid,
+ InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL);
+
return;
}
/*
+ * If conflict detection is enabled, check whether the local
+ * tuple was modified by a different origin. If detected,
+ * report the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
+ InvalidOid, localxmin, localorigin,
+ localts, NULL);
+
+ /*
* Apply the update to the local tuple, putting the result in
* remoteslot_part.
*/
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 2b02148..0f72d28 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4760,6 +4760,7 @@ getSubscriptions(Archive *fout)
int i_suboriginremotelsn;
int i_subenabled;
int i_subfailover;
+ int i_subdetectconflict;
int i,
ntups;
@@ -4832,11 +4833,17 @@ getSubscriptions(Archive *fout)
if (fout->remoteVersion >= 170000)
appendPQExpBufferStr(query,
- " s.subfailover\n");
+ " s.subfailover,\n");
else
appendPQExpBuffer(query,
- " false AS subfailover\n");
+ " false AS subfailover,\n");
+ if (fout->remoteVersion >= 170000)
+ appendPQExpBufferStr(query,
+ " s.subdetectconflict\n");
+ else
+ appendPQExpBuffer(query,
+ " false AS subdetectconflict\n");
appendPQExpBufferStr(query,
"FROM pg_subscription s\n");
@@ -4875,6 +4882,7 @@ getSubscriptions(Archive *fout)
i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn");
i_subenabled = PQfnumber(res, "subenabled");
i_subfailover = PQfnumber(res, "subfailover");
+ i_subdetectconflict = PQfnumber(res, "subdetectconflict");
subinfo = pg_malloc(ntups * sizeof(SubscriptionInfo));
@@ -4921,6 +4929,8 @@ getSubscriptions(Archive *fout)
pg_strdup(PQgetvalue(res, i, i_subenabled));
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ subinfo[i].subdetectconflict =
+ pg_strdup(PQgetvalue(res, i, i_subdetectconflict));
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5161,6 +5171,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (strcmp(subinfo->subfailover, "t") == 0)
appendPQExpBufferStr(query, ", failover = true");
+ if (strcmp(subinfo->subdetectconflict, "t") == 0)
+ appendPQExpBufferStr(query, ", detect_conflict = true");
+
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4b2e587..bbd7cbe 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -671,6 +671,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ char *subdetectconflict;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f2..fef1ad0 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false};
if (pset.sversion < 100000)
{
@@ -6607,6 +6607,10 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subfailover AS \"%s\"\n",
gettext_noop("Failover"));
+ if (pset.sversion >= 170000)
+ appendPQExpBuffer(&buf,
+ ", subdetectconflict AS \"%s\"\n",
+ gettext_noop("Detect conflict"));
appendPQExpBuffer(&buf,
", subsynccommit AS \"%s\"\n"
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index 0244694..1c416cf 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1946,9 +1946,10 @@ psql_completion(const char *text, int start, int end)
COMPLETE_WITH("(", "PUBLICATION");
/* ALTER SUBSCRIPTION <name> SET ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SET", "("))
- COMPLETE_WITH("binary", "disable_on_error", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ COMPLETE_WITH("binary", "detect_conflict", "disable_on_error",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* ALTER SUBSCRIPTION <name> SKIP ( */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) && TailMatches("SKIP", "("))
COMPLETE_WITH("lsn");
@@ -3357,9 +3358,10 @@ psql_completion(const char *text, int start, int end)
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
- "disable_on_error", "enabled", "failover", "origin",
- "password_required", "run_as_owner", "slot_name",
- "streaming", "synchronous_commit", "two_phase");
+ "detect_conflict", "disable_on_error", "enabled",
+ "failover", "origin", "password_required",
+ "run_as_owner", "slot_name", "streaming",
+ "synchronous_commit", "two_phase");
/* CREATE TRIGGER --- is allowed inside CREATE SCHEMA, so use TailMatches */
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0aa14ec..17daf11 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -98,6 +98,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool subdetectconflict; /* True if replication should perform
+ * conflict detection */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -151,6 +154,7 @@ typedef struct Subscription
* (i.e. the main slot and the table sync
* slots) in the upstream database are enabled
* to be synchronized to the standbys. */
+ bool detectconflict; /* True if conflict detection is enabled */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
new file mode 100644
index 0000000..3a7260d
--- /dev/null
+++ b/src/include/replication/conflict.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ * conflict.h
+ * Exports for conflict detection and log
+ *
+ * Copyright (c) 2012-2024, PostgreSQL Global Development Group
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef CONFLICT_H
+#define CONFLICT_H
+
+#include "access/xlogdefs.h"
+#include "executor/tuptable.h"
+#include "nodes/execnodes.h"
+#include "utils/relcache.h"
+#include "utils/timestamp.h"
+
+/*
+ * Conflict types that could be encountered when applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_DIFFER,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_DIFFER,
+} ConflictType;
+
+extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
+ RepOriginId *localorigin, TimestampTz *localts);
+extern void ReportApplyConflict(int elevel, ConflictType type,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot);
+extern void InitConflictIndexes(ResultRelInfo *relInfo);
+
+#endif
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..cc9337c 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/12345
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist2 | 0/0
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+------------------------------+----------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | f | local | dbname=regress_doesnotexist2 | 0/0
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +409,42 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+ERROR: detect_conflict requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | t | off | dbname=regress_doesnotexist | 0/0
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Detect conflict | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+-----------------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..5c740fd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -287,6 +287,21 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - detect_conflict must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, detect_conflict = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (detect_conflict = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- let's do some tests with pg_create_subscription rather than superuser
SET SESSION AUTHORIZATION regress_subscription_user3;
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 471e981..78c0307 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -331,11 +331,12 @@ is( $result, qq(1|bar
2|baz),
'update works with REPLICA IDENTITY FULL and a primary key');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber->append_conf('postgresql.conf', "log_min_messages = debug1");
-$node_subscriber->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub SET (detect_conflict = true)"
+);
$node_subscriber->safe_psql('postgres', "DELETE FROM tab_full_pk");
@@ -352,10 +353,10 @@ $node_publisher->wait_for_catchup('tap_sub');
my $logfile = slurp_file($node_subscriber->logfile, $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation "tab_full_pk"/,
+ qr/conflict update_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be updated./m,
'update target row is missing');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab_full_pk"/,
+ qr/conflict delete_missing detected on relation "public.tab_full_pk".*\n.*DETAIL:.* Did not find the row to be deleted./m,
'delete target row is missing');
$node_subscriber->append_conf('postgresql.conf',
diff --git a/src/test/subscription/t/013_partition.pl b/src/test/subscription/t/013_partition.pl
index 2958052..7a66a06 100644
--- a/src/test/subscription/t/013_partition.pl
+++ b/src/test/subscription/t/013_partition.pl
@@ -343,12 +343,12 @@ $result =
$node_subscriber2->safe_psql('postgres', "SELECT a FROM tab1 ORDER BY 1");
is($result, qq(), 'truncate of tab1 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = true)"
+);
$node_publisher->safe_psql('postgres',
"INSERT INTO tab1 VALUES (1, 'foo'), (4, 'bar'), (10, 'baz')");
@@ -372,21 +372,21 @@ $node_publisher->wait_for_catchup('sub2');
my $logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab1_2_2"/,
+ qr/conflict update_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_1"/,
+ qr/conflict delete_missing detected on relation "public.tab1_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_2_2"/,
+ qr/conflict delete_missing detected on relation "public.tab1_2_2".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_2_2');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab1_def"/,
+ qr/conflict delete_missing detected on relation "public.tab1_def".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab1_def');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub1 SET (detect_conflict = false)"
+);
# Tests for replication using root table identity and schema
@@ -773,12 +773,12 @@ pub_tab2|3|yyy
pub_tab2|5|zzz
xxx_c|6|aaa), 'inserts into tab2 replicated');
-# Check that subscriber handles cases where update/delete target tuple
-# is missing. We have to look for the DEBUG1 log messages about that,
-# so temporarily bump up the log verbosity.
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = debug1");
-$node_subscriber1->reload;
+# To check that subscriber handles cases where update/delete target tuple
+# is missing, detect_conflict is temporarily enabled to log conflicts
+# related to missing tuples.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = true)"
+);
$node_subscriber1->safe_psql('postgres', "DELETE FROM tab2");
@@ -796,15 +796,35 @@ $node_publisher->wait_for_catchup('sub2');
$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
ok( $logfile =~
- qr/logical replication did not find row to be updated in replication target relation's partition "tab2_1"/,
+ qr/conflict update_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be updated./,
'update target row is missing in tab2_1');
ok( $logfile =~
- qr/logical replication did not find row to be deleted in replication target relation "tab2_1"/,
+ qr/conflict delete_missing detected on relation "public.tab2_1".*\n.*DETAIL:.* Did not find the row to be deleted./,
'delete target row is missing in tab2_1');
-$node_subscriber1->append_conf('postgresql.conf',
- "log_min_messages = warning");
-$node_subscriber1->reload;
+# Enable the track_commit_timestamp to detect the conflict when attempting
+# to update a row that was previously modified by a different origin.
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = on');
+$node_subscriber1->restart;
+
+$node_subscriber1->safe_psql('postgres', "INSERT INTO tab2 VALUES (3, 'yyy')");
+$node_publisher->safe_psql('postgres',
+ "UPDATE tab2 SET b = 'quux' WHERE a = 3");
+
+$node_publisher->wait_for_catchup('sub_viaroot');
+
+$logfile = slurp_file($node_subscriber1->logfile(), $log_location);
+ok( $logfile =~
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/,
+ 'updating a tuple that was modified by a different origin');
+
+# The remaining tests no longer test conflict detection.
+$node_subscriber1->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_viaroot SET (detect_conflict = false)"
+);
+
+$node_subscriber1->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_subscriber1->restart;
# Test that replication continues to work correctly after altering the
# partition of a partitioned target table.
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 0ab57a4..496a3c6 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -30,7 +30,7 @@ sub test_skip_lsn
# ERROR with its CONTEXT when retrieving this information.
my $contents = slurp_file($node_subscriber->logfile, $offset);
$contents =~
- qr/duplicate key value violates unique constraint "tbl_pkey".*\n.*DETAIL:.*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
+ qr/conflict insert_exists detected on relation "public.tbl".*\n.*DETAIL:.* Key \(i\)=\(1\) already exists in unique index "tbl_pkey", which was modified by origin \d+ in transaction \d+ at .*\n.*CONTEXT:.* for replication target relation "public.tbl" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m
or die "could not get error-LSN";
my $lsn = $1;
@@ -83,6 +83,7 @@ $node_subscriber->append_conf(
'postgresql.conf',
qq[
max_prepared_transactions = 10
+track_commit_timestamp = on
]);
$node_subscriber->start;
@@ -109,7 +110,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub FOR TABLE tbl");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on)"
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
# Initial synchronization failure causes the subscription to be disabled.
diff --git a/src/test/subscription/t/030_origin.pl b/src/test/subscription/t/030_origin.pl
index 056561f..8c929c0 100644
--- a/src/test/subscription/t/030_origin.pl
+++ b/src/test/subscription/t/030_origin.pl
@@ -27,9 +27,14 @@ my $stderr;
my $node_A = PostgreSQL::Test::Cluster->new('node_A');
$node_A->init(allows_streaming => 'logical');
$node_A->start;
+
# node_B
my $node_B = PostgreSQL::Test::Cluster->new('node_B');
$node_B->init(allows_streaming => 'logical');
+
+# Enable the track_commit_timestamp to detect the conflict when attempting to
+# update a row that was previously modified by a different origin.
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = on');
$node_B->start;
# Create table on node_A
@@ -140,6 +145,44 @@ is($result, qq(),
);
###############################################################################
+# Check that the conflict can be detected when attempting to update or
+# delete a row that was previously modified by a different source.
+###############################################################################
+
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = true);
+ DELETE FROM tab;");
+
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (32);");
+
+# The update should update the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "UPDATE tab SET a = 33 WHERE a = 32;");
+
+$node_B->wait_for_log(
+ qr/Updating a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_B->safe_psql('postgres', "DELETE FROM tab;");
+$node_A->safe_psql('postgres', "INSERT INTO tab VALUES (33);");
+
+# The delete should remove the row on node B that was inserted by node A.
+$node_C->safe_psql('postgres', "DELETE FROM tab WHERE a = 33;");
+
+$node_B->wait_for_log(
+ qr/Deleting a row that was modified by a different origin [0-9]+ in transaction [0-9]+ at .*/
+);
+
+$node_A->wait_for_catchup($subname_BA);
+$node_B->wait_for_catchup($subname_AB);
+
+# The remaining tests no longer test conflict detection.
+$node_B->safe_psql('postgres',
+ "ALTER SUBSCRIPTION $subname_BC SET (detect_conflict = false);");
+
+$node_B->append_conf('postgresql.conf', 'track_commit_timestamp = off');
+$node_B->restart;
+
+###############################################################################
# Specifying origin = NONE indicates that the publisher should only replicate the
# changes that are generated locally from node_B, but in this case since the
# node_B is also subscribing data from node_A, node_B can have remotely
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3deb611..ae1db1a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -467,6 +467,7 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictType
ConnCacheEntry
ConnCacheKey
ConnParams
--
1.8.3.1
v8-0003-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v8-0003-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 9bf4d33a053b906b8a3329805d4482e2a7fdd548 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Wed, 31 Jul 2024 03:22:24 -0400
Subject: [PATCH v8 3/4] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: remote_apply, keep_local, error
- For UPDATE conflicts:
- update_differ: remote_apply, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: remote_apply, keep_local, error
---
src/backend/executor/execReplication.c | 56 ++-
src/backend/replication/logical/conflict.c | 207 ++++++--
src/backend/replication/logical/worker.c | 366 ++++++++++----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 13 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/029_on_error.pl | 9 +
src/test/subscription/t/034_conflict_resolver.pl | 576 +++++++++++++++++++++++
8 files changed, 1104 insertions(+), 129 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 0680bc8..fb96bb7 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -590,8 +590,9 @@ ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleCommitTs(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(ERROR, type, resultRelInfo->ri_RelationDesc, uniqueidx,
- xmin, origin, committs, conflictslot);
+ ReportApplyConflict(type, CR_ERROR, resultRelInfo->ri_RelationDesc,
+ uniqueidx, xmin, origin, committs,
+ conflictslot, false);
}
}
}
@@ -604,7 +605,8 @@ ReCheckConflictIndexes(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -640,11 +642,55 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleCommitTs(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ &apply_remote, NULL, subid);
+
+ ReportApplyConflict(CT_INSERT_EXISTS, resolver, rel,
+ uniqueidx, xmin, origin, committs,
+ *conflictslot, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 720e923..9ac707c 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -26,13 +26,17 @@
#include "catalog/pg_subscription_conflict_d.h"
#include "catalog/pg_inherits.h"
#include "commands/defrem.h"
+#include "executor/executor.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "replication/origin.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
#include "utils/rel.h"
+#include "utils/syscache.h"
const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -93,11 +97,13 @@ const int ConflictTypeDefaultResolvers[] = {
};
static char *build_index_value_desc(Oid indexoid, TupleTableSlot *conflictslot);
-static int errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin,
+static int errdetail_apply_conflict(ConflictType type,
+ ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
RepOriginId localorigin,
TimestampTz localts,
- TupleTableSlot *conflictslot);
+ TupleTableSlot *conflictslot,
+ bool apply_remote);
/*
* Get the xmin and commit timestamp data (origin and timestamp) associated
@@ -132,22 +138,32 @@ GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * Report a conflict when applying remote changes.
+ * Report conflict and resolution applied while applying remote changes.
*/
void
-ReportApplyConflict(int elevel, ConflictType type, Relation localrel,
- Oid conflictidx, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts,
- TupleTableSlot *conflictslot)
+ReportApplyConflict(ConflictType type, ConflictResolver resolver,
+ Relation localrel, Oid conflictidx,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, TupleTableSlot *conflictslot,
+ bool apply_remote)
{
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
ereport(elevel,
errcode(ERRCODE_INTEGRITY_CONSTRAINT_VIOLATION),
- errmsg("conflict %s detected on relation \"%s.%s\"",
+ errmsg("conflict %s detected on relation \"%s.%s\". Resolution: %s",
ConflictTypeNames[type],
get_namespace_name(RelationGetNamespace(localrel)),
- RelationGetRelationName(localrel)),
- errdetail_apply_conflict(type, conflictidx, localxmin, localorigin,
- localts, conflictslot));
+ RelationGetRelationName(localrel),
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(type, resolver, conflictidx, localxmin, localorigin,
+ localts, conflictslot, apply_remote));
}
/*
@@ -182,13 +198,21 @@ InitConflictIndexes(ResultRelInfo *relInfo)
}
/*
- * Add an errdetail() line showing conflict detail.
+ * Add an errdetail() line showing conflict and resolution details.
*/
static int
-errdetail_apply_conflict(ConflictType type, Oid conflictidx,
- TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot)
+errdetail_apply_conflict(ConflictType type, ConflictResolver resolver,
+ Oid conflictidx, TransactionId localxmin,
+ RepOriginId localorigin, TimestampTz localts,
+ TupleTableSlot *conflictslot, bool apply_remote)
{
+ char *applymsg;
+
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
switch (type)
{
case CT_INSERT_EXISTS:
@@ -202,27 +226,43 @@ errdetail_apply_conflict(ConflictType type, Oid conflictidx,
char *index_value = build_index_value_desc(conflictidx,
conflictslot);
- if (index_value && localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
- index_value, get_rel_name(conflictidx), localorigin,
- localxmin, timestamptz_to_str(localts));
- else if (index_value && !localts)
- return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
- index_value, get_rel_name(conflictidx), localxmin);
+ if (resolver == CR_ERROR)
+ {
+ if (index_value && localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified by origin %u in transaction %u at %s.",
+ index_value, get_rel_name(conflictidx), localorigin,
+ localxmin, timestamptz_to_str(localts));
+ else if (index_value && !localts)
+ return errdetail("Key %s already exists in unique index \"%s\", which was modified in transaction %u.",
+ index_value, get_rel_name(conflictidx), localxmin);
+ else
+ return errdetail("Key already exists in unique index \"%s\".",
+ get_rel_name(conflictidx));
+ }
else
- return errdetail("Key already exists in unique index \"%s\".",
- get_rel_name(conflictidx));
+ return errdetail("Key already exists, %s", applymsg);
}
case CT_UPDATE_DIFFER:
- return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Updating a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
case CT_UPDATE_MISSING:
- return errdetail("Did not find the row to be updated.");
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update.");
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ return errdetail("Did not find the row to be updated. UPDATE can not be converted to INSERT, hence ERROR out.");
+ else if (apply_remote)
+ return errdetail("Did not find the row to be updated. Convert UPDATE to INSERT and %s",
+ applymsg);
+ else
+ return errdetail("Did not find the row to be updated, %s",
+ applymsg);
case CT_DELETE_MISSING:
return errdetail("Did not find the row to be deleted.");
case CT_DELETE_DIFFER:
- return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s.",
- localorigin, localxmin, timestamptz_to_str(localts));
+ return errdetail("Deleting a row that was modified by a different origin %u in transaction %u at %s, %s",
+ localorigin, localxmin,
+ timestamptz_to_str(localts), applymsg);
}
return 0; /* silence compiler warning */
@@ -340,6 +380,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
+/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
@@ -499,3 +602,47 @@ RemoveSubscriptionConflictBySubid(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_REMOTE_APPLY:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fc3f80e..36172b2 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2428,10 +2430,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2454,9 +2456,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2466,7 +2472,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ if (MySubscription->detectconflict)
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MyLogicalRepWorker->subid);
+ else
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, NULL, InvalidOid);
+
+
+ /* Apply remote tuple by converting INSERT to UPDATE */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2648,6 +2683,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, MySubscription->detectconflict);
@@ -2671,38 +2708,71 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/*
* If conflict detection is enabled, check whether the local tuple was
- * modified by a different origin. If detected, report the conflict.
+ * modified by a different origin. If detected, report the conflict
+ * and configured resolver.
*/
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ /*
+ * Apply the change if configured resolver is in favor of that, else
+ * ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- if (MySubscription->detectconflict)
- InitConflictIndexes(relinfo);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ if (MySubscription->detectconflict)
+ InitConflictIndexes(relinfo);
+
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
+ }
}
/* Cleanup. */
@@ -2815,6 +2885,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2836,24 +2908,44 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (MySubscription->detectconflict &&
GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_DELETE_DIFFER, localrel, InvalidOid,
- localxmin, localorigin, localts, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_DELETE_DIFFER, resolver, localrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
+
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_DELETE_MISSING, localrel, InvalidOid,
- InvalidTransactionId, InvalidRepOriginId, 0, NULL);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(CT_DELETE_MISSING, resolver, localrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+ }
}
/* Cleanup. */
@@ -2986,19 +3078,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3023,6 +3117,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3032,38 +3129,81 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
if (!found)
{
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
if (MySubscription->detectconflict)
- ReportApplyConflict(LOG, CT_UPDATE_MISSING,
- partrel, InvalidOid,
- InvalidTransactionId,
- InvalidRepOriginId, 0, NULL);
-
- return;
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MyLogicalRepWorker->subid);
+
+ ReportApplyConflict(CT_UPDATE_MISSING, resolver, partrel,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, NULL, apply_remote);
+
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
+ }
}
+ else
+ {
+ /*
+ * The tuple to be updated is found. If conflict detection
+ * is enabled, check whether the local tuple was modified
+ * by a different origin. If detected, report and resolve
+ * the conflict.
+ */
+ if (MySubscription->detectconflict &&
+ GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MyLogicalRepWorker->subid);
- /*
- * If conflict detection is enabled, check whether the local
- * tuple was modified by a different origin. If detected,
- * report the conflict.
- */
- if (MySubscription->detectconflict &&
- GetTupleCommitTs(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
- ReportApplyConflict(LOG, CT_UPDATE_DIFFER, partrel,
- InvalidOid, localxmin, localorigin,
- localts, NULL);
+ ReportApplyConflict(CT_UPDATE_DIFFER, resolver, partrel,
+ InvalidOid, localxmin, localorigin, localts,
+ NULL, apply_remote);
+ }
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple but conflict detection is
+ * OFF
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict detection is ON
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
/*
* Does the updated tuple still satisfy the current
@@ -3073,27 +3213,59 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- EPQState epqstate;
-
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
- ExecOpenIndices(partrelinfo, false);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
- ExecCloseIndices(partrelinfo);
- EvalPlanQualEnd(&epqstate);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. conflict detection is
+ * OFF and found a tuple 2. conflict detection is ON,
+ * update_differ conflict is detected for the found
+ * tuple and the resolver is in favour of applying the
+ * update.
+ */
+
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ ExecOpenIndices(partrelinfo, false);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ EvalPlanQualEnd(&epqstate);
+
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * If conflict detection is OFF, proceed by always applying
+ * the update (as 'apply_remote' is by default true). If
+ * conflict detection is ON, 'apply_remote' can be OFF as well
+ * if the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3134,10 +3306,16 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- /* DELETE old tuple found in the old partition. */
- apply_handle_delete_internal(edata, partrelinfo,
- localslot,
- part_entry->localindexoid);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ apply_handle_delete_internal(edata, partrelinfo,
+ localslot,
+ part_entry->localindexoid);
/* INSERT new tuple into the new partition. */
@@ -3153,19 +3331,25 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 9770752..7d4a698 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -655,7 +657,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c102f74..2793ad6 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -12,6 +12,8 @@
#include "access/xlogdefs.h"
#include "executor/tuptable.h"
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/relcache.h"
#include "utils/timestamp.h"
@@ -82,10 +84,11 @@ typedef struct ConflictTypeResolver
extern bool GetTupleCommitTs(TupleTableSlot *localslot, TransactionId *xmin,
RepOriginId *localorigin, TimestampTz *localts);
-extern void ReportApplyConflict(int elevel, ConflictType type,
+extern void ReportApplyConflict(ConflictType type, ConflictResolver resolver,
Relation localrel, Oid conflictidx,
TransactionId localxmin, RepOriginId localorigin,
- TimestampTz localts, TupleTableSlot *conflictslot);
+ TimestampTz localts, TupleTableSlot *conflictslot,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -95,5 +98,11 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
+extern bool CanCreateFullTuple(Relation localrel, LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7..00ade29 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/029_on_error.pl b/src/test/subscription/t/029_on_error.pl
index 496a3c6..e6f07fa 100644
--- a/src/test/subscription/t/029_on_error.pl
+++ b/src/test/subscription/t/029_on_error.pl
@@ -113,6 +113,11 @@ $node_subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (disable_on_error = true, streaming = on, two_phase = on, detect_conflict = on)"
);
+# Set 'ERROR' conflict resolver for 'insert_exist' conflict type
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=error)");
+
+
# Initial synchronization failure causes the subscription to be disabled.
$node_subscriber->poll_query_until('postgres',
"SELECT subenabled = false FROM pg_catalog.pg_subscription WHERE subname = 'sub'"
@@ -177,6 +182,10 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, "0",
"check all prepared transactions are resolved on the subscriber");
+# Reset conflict resolver for 'insert_exist' conflict type to default.
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub CONFLICT RESOLVER (insert_exists=remote_apply)");
+
$node_subscriber->stop;
$node_publisher->stop;
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000..58751c5
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,576 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on);");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_differ|remote_apply
+delete_missing|skip
+insert_exists|remote_apply
+update_differ|remote_apply
+update_exists|remote_apply
+update_missing|apply_or_skip),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'remote_apply' for 'insert_exists'
+############################################
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict insert_exists detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#########################################
+# Test 'remote_apply' for 'delete_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+#########################################
+# Test 'keep_local' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict delete_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'remote_apply' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_differ detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Did not find the row to be updated. UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (detect_conflict = on)");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict update_missing detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+done_testing();
--
1.8.3.1
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1]/messages/by-id/CAA4eK1+Kb4i64cRD7MCUGkABNzBgQkP8vr5t01N+L_8GtwPgcA@mail.gmail.com. The detect_conflict option has
been removed, and conflict detection is now enabled by default. This
change required the following updates in resolver patches:
patch-0001:
- Removed dependency on the detect_conflict option. Now, default
conflict resolvers are set on CREATE SUBSCRIPTION if no values are
provided.
- To keep the behavior unchanged, the default resolvers are now set as -
insert_exists = error
update_exists = error
update_differ = apply_remote
update_missing = skip
delete_missing = skip
delete_differ = apply_remote
- Added documentation for conflict resolvers.
patch-0002:
- Removed dependency on the detect_conflict option.
- Updated test cases in 034_conflict_resolver.pl to reflect new
default resolvers and the removal of the detect_conflict option.
patch-0003:
- Implemented resolver for the update_exists conflict type. Supported
resolvers are: apply_remote, keep_local, error.
*The timestamp-based resolution patch is not yet rebased due to its
dependency on the detect_conflict option for handling two-phase and
parallel apply-worker workflows. The behavior needs to be reassessed.
To Do:
patch-0001: Add support for pgdump.
patch-0002 and 0003:
- Optimize by avoiding the pre-scan for conflicts in insert_exists
and update_exists when the resolver favors error or skips applying
remote changes.
- Improve the current recursive method used for multiple key conflict
resolution in update_exists.
Thanks Ajin for working on the docs.
[1]: /messages/by-id/CAA4eK1+Kb4i64cRD7MCUGkABNzBgQkP8vr5t01N+L_8GtwPgcA@mail.gmail.com
--
Thanks,
Nisha
Attachments:
v9-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchapplication/octet-stream; name=v9-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREATE.patchDownload
From 8388df6e8ac066b655a8ffbc502acc9ea9c071a9 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 21 Aug 2024 13:50:52 +0530
Subject: [PATCH v9 1/3] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
---
doc/src/sgml/logical-replication.sgml | 89 +----
doc/src/sgml/ref/alter_subscription.sgml | 12 +
doc/src/sgml/ref/create_subscription.sgml | 167 ++++++++++
src/backend/commands/subscriptioncmds.c | 55 ++++
src/backend/parser/gram.y | 25 +-
src/backend/replication/logical/conflict.c | 308 ++++++++++++++++++
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 55 ++++
src/include/nodes/parsenodes.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 61 ++++
src/test/regress/sql/subscription.sql | 32 ++
15 files changed, 778 insertions(+), 84 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 885a2d70ae..f3fb53668c 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1571,7 +1571,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1585,85 +1585,14 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered in the following <firstterm>conflict</firstterm>
- cases:
- <variablelist>
- <varlistentry>
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_differ</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currenly, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_differ</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currenly, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ Additional logging is triggered for specific <literal>conflict_resolvers</literal>.
+ Users can also configure <literal>conflict_types</literal> while creating
+ the subscription.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for details on conflict_types and conflict_resolvers.
+
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the log.
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d007..e7b39f2000
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
</synopsis>
</refsynopsisdiv>
@@ -345,6 +346,17 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d9421..cacce6979e
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -25,6 +25,7 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,172 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-differ">
+ <term><literal>update_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-differ">
+ <term><literal>delete_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c464ae..fe7e3ae51c 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -28,6 +28,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +38,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -439,6 +441,32 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
}
+/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+
+ if (!stmtresolvers)
+ return;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+
+ /* validate the conflict type and resolver */
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* update the corresponding resolver for the given conflict type */
+ resolvers[type].resolver = defGetString(defel);
+ }
+}
+
/*
* Add publication names from the list to a string.
*/
@@ -583,6 +611,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
/*
* Parse and check options.
@@ -597,6 +626,12 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /*
+ * Parse and check conflict resolvers. Initialize with default values
+ */
+ SetDefaultResolvers(conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -726,6 +761,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
/*
* Connect to remote side to execute requested commands and fetch table
* info.
@@ -1581,6 +1619,20 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1884,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c3f25582c3..628feefd92 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,7 +426,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -772,7 +772,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8806,6 +8806,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10740,7 +10745,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10748,6 +10753,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10854,6 +10860,17 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+
;
/*****************************************************************************
@@ -17778,6 +17795,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18408,6 +18426,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 0bc7959980..ad79f816dd 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,12 +15,26 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -31,6 +45,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_DIFFER] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -486,3 +549,248 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *CTR = NULL;
+ List *res = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ CTR = palloc(sizeof(ConflictTypeResolver));
+ CTR->conflict_type = defel->defname;
+ CTR->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+
+ res = lappend(res, CTR);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolver for a conflict type in
+ * pg_subscription_conflict system catalog
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int type;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ for (type = 0; type < resolvers_cnt; type++)
+ {
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[type].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[type].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..c8b37c2bef
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTSUBOID, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 85a62b538e..2e1eefc3a7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4203,6 +4203,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4215,6 +4216,7 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4225,6 +4227,7 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index bb191b1f46..7bffc9401b 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -376,6 +376,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 02cb84da7e..fac3f8335e 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -42,6 +42,47 @@ typedef enum
*/
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -54,5 +95,13 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictBySubid(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..fbf355c20f 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -399,6 +399,67 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | apply_remote
+ delete_missing | skip
+ insert_exists | error
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | keep_local
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_differ = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | apply_remote
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..b7ccc02a51 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,38 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_differ = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
--
2.34.1
v9-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v9-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 16b986f8f8544f090d46023f4b7a904b777e7e36 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 21 Aug 2024 08:51:52 +0530
Subject: [PATCH v9 2/3] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_differ: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 57 +-
src/backend/replication/logical/conflict.c | 210 +++++--
src/backend/replication/logical/worker.c | 357 ++++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 14 +-
src/test/subscription/meson.build | 1 +
.../subscription/t/034_conflict_resolver.pl | 581 ++++++++++++++++++
7 files changed, 1080 insertions(+), 145 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc962..c13cfcb77c 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,9 +550,9 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
- searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ ReportApplyConflict(estate, resultRelInfo, type,
+ CR_ERROR, searchslot, conflictslot, remoteslot,
+ uniqueidx, xmin, origin, committs, false);
}
}
}
@@ -565,7 +565,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -601,11 +602,55 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS,
+ &apply_remote, NULL, subid);
+
+ ReportApplyConflict(estate, resultRelInfo, CT_INSERT_EXISTS,
+ resolver, NULL, *conflictslot, slot,
+ uniqueidx, xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0)
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, false,
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index ad79f816dd..df92c616aa 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -31,6 +31,7 @@
#include "replication/conflict.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
@@ -98,12 +99,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -146,8 +149,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -166,26 +169,36 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -253,17 +266,25 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -274,13 +295,13 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts), applymsg);
/*
* The origin that modified this row has been removed. This
@@ -290,47 +311,55 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_DIFFER:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin, timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts), applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"), applymsg);
break;
case CT_DELETE_DIFFER:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin, timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts), applymsg);
break;
@@ -634,6 +663,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
@@ -794,3 +886,47 @@ RemoveSubscriptionConflictBySubid(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 38c2895307..be68445565 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,35 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If conflict is detected and resolver is in favor of applying the remote
+ * changes, then, apply remote tuple by converting INSERT to UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2703,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2726,44 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_DIFFER,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
- }
-
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_DIFFER,
+ resolver, remoteslot, localslot, newslot,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ } /* Apply the change if configured resolver is
+ * in favor of that, else ignore the remote
+ * update. */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2773,27 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING,
+ resolver, remoteslot, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot,
+ newtup, relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2906,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2868,26 +2928,41 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_DIFFER,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
+ ReportApplyConflict(estate, relinfo, CT_DELETE_DIFFER,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3020,19 +3095,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3058,6 +3135,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3070,47 +3150,84 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ if (apply_remote)
+ {
+ /*
+ * Resolver is in favour of applying the remote
+ * changes. Prepare the slot for the INSERT.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_DIFFER,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_DIFFER,
+ resolver, remoteslot_part, localslot, newslot,
+ InvalidOid, localxmin, localorigin,
+ localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and resolver is in favor of
+ * applying the change when conflict is detected
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3122,23 +3239,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_differ conflict is
+ * detected for the found tuple and the resolver is in
+ * favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3179,12 +3325,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3200,22 +3354,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00..f3909759f8 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index fac3f8335e..0ddbfbe9c2 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -88,12 +90,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -103,5 +107,11 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
+extern bool CanCreateFullTuple(Relation localrel, LogicalRepTupleData *newtup);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..6f2ba4c7ba
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,581 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_differ|apply_remote
+delete_missing|skip
+insert_exists|error
+update_differ|apply_remote
+update_exists|error
+update_missing|skip),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists./,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#########################################
+# Test 'apply_remote' for 'delete_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_differ/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+#########################################
+# Test 'keep_local' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_differ/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_differ./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'apply_remote' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_differ/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_differ/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_differ./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v9-0003-Conflict-resolution-for-update_exists-conflict-ty.patchapplication/octet-stream; name=v9-0003-Conflict-resolution-for-update_exists-conflict-ty.patchDownload
From ba2f9e5a933cfb707ca4415537e38128ed0020ca Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Wed, 21 Aug 2024 15:14:02 +0530
Subject: [PATCH v9 3/3] Conflict resolution for 'update_exists' conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 60 +++++++++--
src/backend/replication/logical/worker.c | 74 ++++++++++---
src/include/executor/executor.h | 3 +-
.../subscription/t/032_subscribe_use_index.pl | 2 +-
.../subscription/t/034_conflict_resolver.pl | 100 ++++++++++++++++++
5 files changed, 217 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index c13cfcb77c..56ee179059 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -589,6 +589,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -620,7 +621,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* error here; otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
- slot, &(*conflictslot)))
+ slot, &(*conflictslot), &invalidItemPtr))
{
RepOriginId origin;
TimestampTz committs;
@@ -700,7 +701,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -739,11 +741,55 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /*
+ * If caller has passed non null conflictslot, check all the unique
+ * indexes for potential conflicts. If the configured resolver is in
+ * favour of apply, give the conflicted tuple information in
+ * conflictslot.
+ */
+ if (conflictslot)
+ {
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit
+ * error here; otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx,
+ slot, &(*conflictslot), &searchslot->tts_tid))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+ ConflictResolver resolver;
+ bool apply_remote = false;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS,
+ &apply_remote, NULL, subid);
+
+ ReportApplyConflict(estate, resultRelInfo, CT_UPDATE_EXISTS,
+ resolver, NULL, *conflictslot, slot,
+ uniqueidx, xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return;
+ }
+ }
+ }
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
- conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
-
if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None))
recheckIndexes = ExecInsertIndexTuples(resultRelInfo,
slot, estate, true,
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index be68445565..a940d019d7 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2516,8 +2517,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup, rel_entry->localindexoid,
+ true, conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2669,7 +2671,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2694,14 +2697,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2709,10 +2711,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2750,6 +2753,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* update. */
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2762,7 +2767,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot, MySubscription->oid);
+
+ /* UPDATE the conflicting tuple */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* update the conflicting row */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3253,6 +3279,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* detected for the found tuple and the resolver is in
* favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3260,7 +3288,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /* UPDATE the conflicting tuple */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* update the conflicting row */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup, part_entry->localindexoid, true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f3909759f8..291d5dde36 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/032_subscribe_use_index.pl b/src/test/subscription/t/032_subscribe_use_index.pl
index cc999e33c3..b7f30e32cd 100644
--- a/src/test/subscription/t/032_subscribe_use_index.pl
+++ b/src/test/subscription/t/032_subscribe_use_index.pl
@@ -455,7 +455,7 @@ $node_publisher->safe_psql('postgres',
# wait until the index is used on the subscriber
$node_publisher->wait_for_catchup($appname);
$node_subscriber->poll_query_until('postgres',
- q{select (idx_scan = 1) from pg_stat_all_indexes where indexrelname = 'test_replica_id_full_idxy';}
+ q{select (idx_scan = 2) from pg_stat_all_indexes where indexrelname = 'test_replica_id_full_idxy';}
)
or die
"Timed out while waiting for check subscriber tap_sub_rep_full updates one row via index";
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 6f2ba4c7ba..0c4d808407 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
--
2.34.1
Import Notes
Reply to msg id not found: CAJpy0uB02LPqAAVuTKAMerMVkmHOSZO3UEEKtaJ7ihkfjYubQ@mail.gmail.com
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].
Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:
1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);
1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);
Earlier the syntax suggested in [1]/messages/by-id/CAA4eK1LhD=C5UwDeKxC_5jK4_ADtM7g+MoFW9qhziSxHbVVfeQ@mail.gmail.com was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';
I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.
~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?
2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS
2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';
The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?
~~
3) Regarding update_exists:
3a)
Currently update_exists resolver patch is kept separate. The reason
being, it performs resolution which will need deletion of multiple
rows. It will be good to discuss if we want to target this in the
first draft. Please see the example:
create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);
Pub: update tab set a=2,b=40,c=70 where a=1;
The above 'update' on pub will result in 'update_exists' on sub and if
resolution is in favour of 'apply', then it will conflict with all the
three local rows of subscriber due to unique constraint present on all
three columns. Thus in order to resolve the conflict, it will have to
delete these 3 rows on sub:
2,20,30
3,40,50
4,60,70
and then update 1,1,1 to 2,40,70.
Just need opinion on if we shall target this in the initial draft.
3b)
If we plan to implement this, we need to work on optimal design where
we can find all the conflicting rows at once and delete those.
Currently the implementation has been done using recursion i.e. find
one conflicting row, then delete it and then next and so on i.e. we
call apply_handle_update_internal() recursively. On initial code
review, I feel it is doable to scan all indexes at once and get
conflicting-tuple-ids in one go and get rid of recursion. It can be
attempted once we decide on 3a.
~~
4)
Now for insert_exists and update_exists, we are doing a pre-scan of
all unique indexes to find conflict. Also there is post-scan to figure
out if the conflicting row is inserted meanwhile. This needs to be
reviewed for optimization. We need to avoid pre-scan wherever
possible. I think the only case for which it can be avoided is
'ERROR'. For the cases where resolver is in favor of remote-apply, we
need to check conflict beforehand to avoid rollback of already
inserted data. And for the case where resolver is in favor of skipping
the change, then too we should know beforehand about the conflict to
avoid heap-insertion and rollback. Thoughts?
~~
5)
Currently we only capture update_missing conflict i.e. we are not
distinguishing between the missing row and the deleted row. We had
discussed this in the past a couple of times. If we plan to target it
in draft 1, I can dig up all old emails and resume discussion on this.
~~
6)
Table-level resolves. There was a suggestion earlier to implement
table-level resolvers. The patch has been implemented to some extent,
it can be completed and posted when we are done reviewing subscription
level resolvers.
~~
[1]: /messages/by-id/CAA4eK1LhD=C5UwDeKxC_5jK4_ADtM7g+MoFW9qhziSxHbVVfeQ@mail.gmail.com
For clock-skew and timestamp based resolution, if needed, I will post
another email for the design items where suggestions are needed.
thanks
Shveta
On Thu, Aug 22, 2024 at 3:44 PM shveta malik <shveta.malik@gmail.com> wrote:
For clock-skew and timestamp based resolution, if needed, I will post
another email for the design items where suggestions are needed.
Please find issues which need some thoughts and approval for
time-based resolution and clock-skew.
1)
Time based conflict resolution and two phase transactions:
Time based conflict resolution (last_update_wins) is the one
resolution which will not result in data-divergence considering
clock-skew is taken care of. But when it comes to two-phase
transactions, it might not be the case. For two-phase transaction, we
do not have commit timestamp when the changes are being applied. Thus
for time-based comparison, initially it was decided to user prepare
timestamp but it may result in data-divergence. Please see the
example at [1]Example of 2pc inconsistency: --------------------------------------------------------- Two nodes, A and B, are subscribed to each other and have identical data. The last_update_wins strategy is configured..
Example at [1]Example of 2pc inconsistency: --------------------------------------------------------- Two nodes, A and B, are subscribed to each other and have identical data. The last_update_wins strategy is configured. is a tricky situation, and thus in the initial draft,
we decided to restrict usage of 2pc and CDR together. The plan is:
a) During Create subscription, if the user has given last_update_wins
resolver for any conflict_type and 'two_phase' is also enabled, we
ERROR out.
b) During Alter subscription, if the user tries to update resolver to
'last_update_wins' but 'two_phase' is enabled, we error out.
Another solution could be to save both prepare_ts and commit_ts. And
when any txn comes for conflict resolution, we first check if
prepare_ts is available, use that else use commit_ts. Availability of
prepare_ts would indicate it was a prepared txn and thus even if it is
committed, we should use prepare_ts for comparison for consistency.
This will have some overhead of storing prepare_ts along with
commit_ts. But if the number of prepared txns are reasonably small,
this overhead should be less.
We currently plan to go with restricting 2pc and last_update_wins
together, unless others have different opinions.
~~
2)
parallel apply worker and conflict-resolution:
As discussed in [2]/messages/by-id/CAFiTN-sf23K=sRsnxw-BKNJqg5P6JXcqXBBkx=EULX8QGSQYaw@mail.gmail.com (see last paragraph in [2]/messages/by-id/CAFiTN-sf23K=sRsnxw-BKNJqg5P6JXcqXBBkx=EULX8QGSQYaw@mail.gmail.com), for streaming of
in-progress transactions by parallel worker, we do not have
commit-timestamp with each change and thus it makes sense to disable
parallel apply worker with CDR. The plan is to not start parallel
apply worker if 'last_update_wins' is configured for any
conflict_type.
~~
3)
parallel apply worker and clock skew management:
Regarding clock-skew management as discussed in [3]/messages/by-id/CAA4eK1+hdMmwEEiMb4z6x7JgQbw1jU2XyP1U7dNObyUe4JQQWg@mail.gmail.com, we will wait for
the local clock to come within tolerable range during 'begin' rather
than before 'commit'. And this wait needs commit-timestamp in the
beginning, thus we plan to restrict starting pa-worker even when
clock-skew related GUCs are configured.
Earlier we had restricted both 2pc and parallel worker worker start
when detect_conflict was enabled, but now since detect_conflict
parameter is removed, we will change the implementation to restrict
all 3 above cases when last_update_wins is configured. When the
changes are done, we will post the patch.
~~
4)
<not related to timestamp and clock skew>
Earlier when 'detect_conflict' was enabled, we were giving WARNING if
'track_commit_timestamp' was not enabled. This was during CREATE and
ALTER subscription. Now with this parameter removed, this WARNING has
also been removed. But I think we need to bring back this WARNING.
Currently default resolvers set may work without
'track_commit_timestamp' but when user gives CONFLICT RESOLVER in
create-sub or alter-sub explicitly making them configured to
non-default values (or say any values, does not matter if few are
defaults), we may still emit this warning to alert user:
2024-07-26 09:14:03.152 IST [195415] WARNING: conflict detection
could be incomplete due to disabled track_commit_timestamp
2024-07-26 09:14:03.152 IST [195415] DETAIL: Conflicts update_differ
and delete_differ cannot be detected, and the origin and commit
timestamp for the local row will not be logged.
Thoughts?
If we emit this WARNING during each resolution, then it may flood our
log files, thus it seems better to emit it during create or alter
subscription instead of during resolution.
~~
[1]: Example of 2pc inconsistency: --------------------------------------------------------- Two nodes, A and B, are subscribed to each other and have identical data. The last_update_wins strategy is configured.
Example of 2pc inconsistency:
---------------------------------------------------------
Two nodes, A and B, are subscribed to each other and have identical
data. The last_update_wins strategy is configured.
Both contain the data: '1, x, node'.
Timeline of Events:
9:00 AM on Node A: A transaction (txn1) is prepared to update the row
to '1, x, nodeAAA'. We'll refer to this as change1 on Node A.
9:01 AM on Node B: An update occurs for the row, changing it to '1, x,
nodeBBB'. This update is then sent to Node A. We'll call this change2
on Node B.
At 9:02 AM:
--Node A: Still holds '1, x, node' because txn1 is not yet committed.
--Node B: Holds '1, x, nodeBBB'.
--Node B receives the prepared transaction from Node A at 9:02 AM and
raises an update_differ conflict.
--Since the local change occurred at 9:01 AM, which is later than the
9:00 AM prepare-timestamp from Node A, Node B retains its local
change.
At 9:05 AM:
--Node A commits the prepared txn1.
--The apply worker on Node A has been waiting to apply the changes
from Node B because the tuple was locked by txn1.
--Once the commit occurs, the apply worker proceeds with the update from Node B.
--When update_differ is triggered, since the 9:05 AM commit-timestamp
from Node A is later than the 9:01 AM commit-timestamp from Node B,
Node A’s update wins.
Final Data on Nodes:
Node A: '1, x, nodeAAA'
Node B: '1, x, nodeBBB'
Despite the last_update_wins resolution, the nodes end up with different data.
The data divergence happened because on node B, we used change1's
prepare_ts (9.00) for comparison; while on node A, we used change1's
commit_ts(9.05) for comparison.
---------------------------------------------------------
[2]: /messages/by-id/CAFiTN-sf23K=sRsnxw-BKNJqg5P6JXcqXBBkx=EULX8QGSQYaw@mail.gmail.com
[3]: /messages/by-id/CAA4eK1+hdMmwEEiMb4z6x7JgQbw1jU2XyP1U7dNObyUe4JQQWg@mail.gmail.com
thanks
Shveta
On Thu, Aug 22, 2024 at 8:15 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?
Hi Shveta,
I felt it would be better to keep the syntax similar to the existing
INSERT ... ON CONFLICT [1]https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT.
I'd suggest a syntax like this:
... ON CONFLICT ['conflict_type'] DO { 'conflict_action' | DEFAULT }
~~~
e.g.
To configure conflict resolvers for the SUBSCRIPTION:
CREATE SUBSCRIPTION subname CONNECTION coninfo PUBLICATION pubname
ON CONFLICT 'conflict_type1' DO 'conflict_action1',
ON CONFLICT 'conflict_type2' DO 'conflict_action2';
Likewise, for ALTER:
ALTER SUBSCRIPTION <subname>
ON CONFLICT 'conflict_type1' DO 'conflict_action1',
ON CONFLICT 'conflict_type2' DO 'conflict_action2';
To RESET all at once:
ALTER SUBSCRIPTION <subname>
ON CONFLICT DO DEFAULT;
And, to RESET one at a time:
ALTER SUBSCRIPTION <subname>
ON CONFLICT 'conflict_type1' DO DEFAULT;
~~~
Although your list format "('conflict_type1' = 'conflict_action1',
'conflict_type2' = 'conflict_action2')" is clear and without
repetition, I predict this terse style could end up being troublesome
because it does not offer much flexibility for whatever the future
might hold for CDR.
e.g. ability to handle the conflict with a user-defined resolver
e.g. ability to handle the conflict conditionally (e.g. with a WHERE clause...)
e.g. ability to handle all conflicts with a common resolver
etc.
~~~~
Advantages of my suggestion:
- Close to existing SQL syntax
- No loss of clarity by removing the word "RESOLVER"
- No requirement for new keyword/s
- The commands now read more like English
- Offers more flexibility for any unknown future requirements
- The setup (via create subscription) and the alter/reset all look the same.
======
[1]: https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICT
Kind Regards,
Peter Smith.
Fujitsu Australia
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1]. The detect_conflict option has
been removed, and conflict detection is now enabled by default. This
change required the following updates in resolver patches:
patch-0001:
- Removed dependency on the detect_conflict option. Now, default
conflict resolvers are set on CREATE SUBSCRIPTION if no values are
provided.
- To keep the behavior unchanged, the default resolvers are now set as -
insert_exists = error
update_exists = error
update_differ = apply_remote
update_missing = skip
delete_missing = skip
delete_differ = apply_remote
- Added documentation for conflict resolvers.patch-0002:
- Removed dependency on the detect_conflict option.
- Updated test cases in 034_conflict_resolver.pl to reflect new
default resolvers and the removal of the detect_conflict option.patch-0003:
- Implemented resolver for the update_exists conflict type. Supported
resolvers are: apply_remote, keep_local, error.
Thanks Nisha for the patches, I was running some tests on
update_exists and found this case wherein it misses to LOG one
conflict out of 3.
create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);
Pub: update tab set a=2,b=40,c=70 where a=1;
Here it logs update_exists conflict and the resolution for Key
(b)=(40) and Key (c)=(70) but misses to LOG first one which is with
Key (a)=(2).
thanks
Shveta
On Mon, Aug 26, 2024 at 7:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Thu, Aug 22, 2024 at 8:15 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?Hi Shveta,
I felt it would be better to keep the syntax similar to the existing
INSERT ... ON CONFLICT [1].I'd suggest a syntax like this:
... ON CONFLICT ['conflict_type'] DO { 'conflict_action' | DEFAULT }
~~~
e.g.
To configure conflict resolvers for the SUBSCRIPTION:
CREATE SUBSCRIPTION subname CONNECTION coninfo PUBLICATION pubname
ON CONFLICT 'conflict_type1' DO 'conflict_action1',
ON CONFLICT 'conflict_type2' DO 'conflict_action2';Likewise, for ALTER:
ALTER SUBSCRIPTION <subname>
ON CONFLICT 'conflict_type1' DO 'conflict_action1',
ON CONFLICT 'conflict_type2' DO 'conflict_action2';To RESET all at once:
ALTER SUBSCRIPTION <subname>
ON CONFLICT DO DEFAULT;And, to RESET one at a time:
ALTER SUBSCRIPTION <subname>
ON CONFLICT 'conflict_type1' DO DEFAULT;
Thanks for the suggestion. The idea looks good to me. But we need to
once check the complexity involved in its implementation in gram.y.
Initial analysis says that it will need something like 'action' which
we have for ALTER TABLE command ([1]https://www.postgresql.org/docs/current/sql-altertable.html) to have these multiple
subcommands implemented. For INSERT case, it is a just a subclause but
for create/alter sub we hill have it multiple times under one command.
Let us review.
Also I would like to know opinion of others on this.
[1]: https://www.postgresql.org/docs/current/sql-altertable.html
Show quoted text
Although your list format "('conflict_type1' = 'conflict_action1',
'conflict_type2' = 'conflict_action2')" is clear and without
repetition, I predict this terse style could end up being troublesome
because it does not offer much flexibility for whatever the future
might hold for CDR.e.g. ability to handle the conflict with a user-defined resolver
e.g. ability to handle the conflict conditionally (e.g. with a WHERE clause...)
e.g. ability to handle all conflicts with a common resolver
etc.~~~~
Advantages of my suggestion:
- Close to existing SQL syntax
- No loss of clarity by removing the word "RESOLVER"
- No requirement for new keyword/s
- The commands now read more like English
- Offers more flexibility for any unknown future requirements
- The setup (via create subscription) and the alter/reset all look the same.======
[1] https://www.postgresql.org/docs/current/sql-insert.html#SQL-ON-CONFLICTKind Regards,
Peter Smith.
Fujitsu Australia
On Thu, Aug 22, 2024 at 3:45 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?~~
3) Regarding update_exists:
3a)
Currently update_exists resolver patch is kept separate. The reason
being, it performs resolution which will need deletion of multiple
rows. It will be good to discuss if we want to target this in the
first draft. Please see the example:create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);Pub: update tab set a=2,b=40,c=70 where a=1;
The above 'update' on pub will result in 'update_exists' on sub and if
resolution is in favour of 'apply', then it will conflict with all the
three local rows of subscriber due to unique constraint present on all
three columns. Thus in order to resolve the conflict, it will have to
delete these 3 rows on sub:2,20,30
3,40,50
4,60,70
and then update 1,1,1 to 2,40,70.Just need opinion on if we shall target this in the initial draft.
3b)
If we plan to implement this, we need to work on optimal design where
we can find all the conflicting rows at once and delete those.
Currently the implementation has been done using recursion i.e. find
one conflicting row, then delete it and then next and so on i.e. we
call apply_handle_update_internal() recursively. On initial code
review, I feel it is doable to scan all indexes at once and get
conflicting-tuple-ids in one go and get rid of recursion. It can be
attempted once we decide on 3a.~~
4)
Now for insert_exists and update_exists, we are doing a pre-scan of
all unique indexes to find conflict. Also there is post-scan to figure
out if the conflicting row is inserted meanwhile. This needs to be
reviewed for optimization. We need to avoid pre-scan wherever
possible. I think the only case for which it can be avoided is
'ERROR'. For the cases where resolver is in favor of remote-apply, we
need to check conflict beforehand to avoid rollback of already
inserted data. And for the case where resolver is in favor of skipping
the change, then too we should know beforehand about the conflict to
avoid heap-insertion and rollback. Thoughts?
+1 to the idea of optimization, but it seems that when the resolver is
set to ERROR, skipping the pre-scan only optimizes the case where no
conflict exists.
If a conflict is found, the apply-worker will error out during the
pre-scan, and no post-scan occurs, so there's no opportunity for
optimization.
However, if no conflict is present, we currently do both pre-scan and
post-scan. Skipping the pre-scan in this scenario could be a
worthwhile optimization, even if it only benefits the no-conflict
case.
--
Thanks,
Nisha
On Thu, Aug 22, 2024 at 3:45 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?
It makes sense to have a RESET on the lines of (a) and (b). At this
stage, we should do minimal in extending the syntax. How about RESET
CONFLICT RESOLVER ALL for (a)?
~~
3) Regarding update_exists:
3a)
Currently update_exists resolver patch is kept separate. The reason
being, it performs resolution which will need deletion of multiple
rows. It will be good to discuss if we want to target this in the
first draft. Please see the example:create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);Pub: update tab set a=2,b=40,c=70 where a=1;
The above 'update' on pub will result in 'update_exists' on sub and if
resolution is in favour of 'apply', then it will conflict with all the
three local rows of subscriber due to unique constraint present on all
three columns. Thus in order to resolve the conflict, it will have to
delete these 3 rows on sub:2,20,30
3,40,50
4,60,70
and then update 1,1,1 to 2,40,70.Just need opinion on if we shall target this in the initial draft.
This case looks a bit complicated. It seems there is no other
alternative than to delete the multiple rows. It is better to create a
separate top-up patch for this and we can discuss in detail about this
once the basic patch is in better shape.
3b)
If we plan to implement this, we need to work on optimal design where
we can find all the conflicting rows at once and delete those.
Currently the implementation has been done using recursion i.e. find
one conflicting row, then delete it and then next and so on i.e. we
call apply_handle_update_internal() recursively. On initial code
review, I feel it is doable to scan all indexes at once and get
conflicting-tuple-ids in one go and get rid of recursion. It can be
attempted once we decide on 3a.
I suggest following the simplest strategy (even if that means calling
the update function recursively) by adding comments on the optimal
strategy. We can optimize it later as well.
~~
4)
Now for insert_exists and update_exists, we are doing a pre-scan of
all unique indexes to find conflict. Also there is post-scan to figure
out if the conflicting row is inserted meanwhile. This needs to be
reviewed for optimization. We need to avoid pre-scan wherever
possible. I think the only case for which it can be avoided is
'ERROR'. For the cases where resolver is in favor of remote-apply, we
need to check conflict beforehand to avoid rollback of already
inserted data. And for the case where resolver is in favor of skipping
the change, then too we should know beforehand about the conflict to
avoid heap-insertion and rollback. Thoughts?
It makes sense to skip the pre-scan wherever possible. Your analysis
sounds reasonable to me.
~~
5)
Currently we only capture update_missing conflict i.e. we are not
distinguishing between the missing row and the deleted row. We had
discussed this in the past a couple of times. If we plan to target it
in draft 1, I can dig up all old emails and resume discussion on this.
This is a separate conflict detection project in itself. I am thinking
about the solution to this problem. We will talk about this in a
separate thread.
~~
6)
Table-level resolves. There was a suggestion earlier to implement
table-level resolvers. The patch has been implemented to some extent,
it can be completed and posted when we are done reviewing subscription
level resolvers.
Yeah, it makes sense to do it after the subscription-level resolution
patch is ready.
--
With Regards,
Amit Kapila.
On Mon, Aug 26, 2024 at 7:28 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Thu, Aug 22, 2024 at 8:15 PM shveta malik <shveta.malik@gmail.com> wrote:
Hi Shveta,
I felt it would be better to keep the syntax similar to the existing
INSERT ... ON CONFLICT [1].I'd suggest a syntax like this:
... ON CONFLICT ['conflict_type'] DO { 'conflict_action' | DEFAULT }
~~~
e.g.
To configure conflict resolvers for the SUBSCRIPTION:
CREATE SUBSCRIPTION subname CONNECTION coninfo PUBLICATION pubname
ON CONFLICT 'conflict_type1' DO 'conflict_action1',
ON CONFLICT 'conflict_type2' DO 'conflict_action2';
One thing that looks odd to me about this is the resolution part of
it. For example, ON CONFLICT 'insert_exists' DO 'keep_local'. The
action part doesn't go well without being explicit that it is a
resolution method. Another variant could be ON CONFLICT
'insert_exists' USE RESOLUTION [METHOD] 'keep_local'.
I think we can keep all these syntax alternatives either in the form
of comments or in the commit message and discuss more on these once we
agree on the solutions to the key design issues pointed out by Shveta.
--
With Regards,
Amit Kapila.
On Mon, Aug 26, 2024 at 2:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Aug 22, 2024 at 3:45 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?It makes sense to have a RESET on the lines of (a) and (b). At this
stage, we should do minimal in extending the syntax. How about RESET
CONFLICT RESOLVER ALL for (a)?
Yes, the syntax looks good.
~~
3) Regarding update_exists:
3a)
Currently update_exists resolver patch is kept separate. The reason
being, it performs resolution which will need deletion of multiple
rows. It will be good to discuss if we want to target this in the
first draft. Please see the example:create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);Pub: update tab set a=2,b=40,c=70 where a=1;
The above 'update' on pub will result in 'update_exists' on sub and if
resolution is in favour of 'apply', then it will conflict with all the
three local rows of subscriber due to unique constraint present on all
three columns. Thus in order to resolve the conflict, it will have to
delete these 3 rows on sub:2,20,30
3,40,50
4,60,70
and then update 1,1,1 to 2,40,70.Just need opinion on if we shall target this in the initial draft.
This case looks a bit complicated. It seems there is no other
alternative than to delete the multiple rows. It is better to create a
separate top-up patch for this and we can discuss in detail about this
once the basic patch is in better shape.
Agreed.
3b)
If we plan to implement this, we need to work on optimal design where
we can find all the conflicting rows at once and delete those.
Currently the implementation has been done using recursion i.e. find
one conflicting row, then delete it and then next and so on i.e. we
call apply_handle_update_internal() recursively. On initial code
review, I feel it is doable to scan all indexes at once and get
conflicting-tuple-ids in one go and get rid of recursion. It can be
attempted once we decide on 3a.I suggest following the simplest strategy (even if that means calling
the update function recursively) by adding comments on the optimal
strategy. We can optimize it later as well.
Sure.
Show quoted text
~~
4)
Now for insert_exists and update_exists, we are doing a pre-scan of
all unique indexes to find conflict. Also there is post-scan to figure
out if the conflicting row is inserted meanwhile. This needs to be
reviewed for optimization. We need to avoid pre-scan wherever
possible. I think the only case for which it can be avoided is
'ERROR'. For the cases where resolver is in favor of remote-apply, we
need to check conflict beforehand to avoid rollback of already
inserted data. And for the case where resolver is in favor of skipping
the change, then too we should know beforehand about the conflict to
avoid heap-insertion and rollback. Thoughts?It makes sense to skip the pre-scan wherever possible. Your analysis
sounds reasonable to me.~~
5)
Currently we only capture update_missing conflict i.e. we are not
distinguishing between the missing row and the deleted row. We had
discussed this in the past a couple of times. If we plan to target it
in draft 1, I can dig up all old emails and resume discussion on this.This is a separate conflict detection project in itself. I am thinking
about the solution to this problem. We will talk about this in a
separate thread.~~
6)
Table-level resolves. There was a suggestion earlier to implement
table-level resolvers. The patch has been implemented to some extent,
it can be completed and posted when we are done reviewing subscription
level resolvers.Yeah, it makes sense to do it after the subscription-level resolution
patch is ready.--
With Regards,
Amit Kapila.
Please find v10 patch-set. Changes are:
1) patch-001:
- Corrected a patch application warning.
- Added support for pg_dump.
- As suggested in pt.4 of [1]/messages/by-id/CAJpy0uA0J8kz2DKU0xbUkUT=rtt=CenpzmUMgYcwms9+zgCuvA@mail.gmail.com: added a warning during CREATE and
ALTER subscription when track_commit_timestamp is OFF.
2) patch-002 & patch-003:
- Reduced code duplication in execReplication.c
- As suggested in pt.4 of [2]/messages/by-id/CAJpy0uBrXZE6LLofX5tc8WOm5F+FNgnQjRLQerOY8cOqqvtrNg@mail.gmail.com: Optimized the pre-scan for
insert_exists and update_exists cases when resolver is set to ERROR.
- Fixed a bug reported by Shveta in [3]/messages/by-id/CAJpy0uCtYweJHwuYdgGWR7iSDUKjqDtA7yoLe+XMWqWnmQhj8g@mail.gmail.com
Thank You Ajin for working on pg_dump support changes.
[1]: /messages/by-id/CAJpy0uA0J8kz2DKU0xbUkUT=rtt=CenpzmUMgYcwms9+zgCuvA@mail.gmail.com
[2]: /messages/by-id/CAJpy0uBrXZE6LLofX5tc8WOm5F+FNgnQjRLQerOY8cOqqvtrNg@mail.gmail.com
[3]: /messages/by-id/CAJpy0uCtYweJHwuYdgGWR7iSDUKjqDtA7yoLe+XMWqWnmQhj8g@mail.gmail.com
Thanks,
Nisha
Attachments:
v10-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v10-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From e3a64e6891414c77683a8cad2fc3d9dbd6775e57 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 22 Aug 2024 10:20:33 +0530
Subject: [PATCH v10 1/3] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 89 +----
doc/src/sgml/ref/alter_subscription.sgml | 12 +
doc/src/sgml/ref/create_subscription.sgml | 167 ++++++++++
src/backend/commands/subscriptioncmds.c | 77 +++++
src/backend/parser/gram.y | 25 +-
src/backend/replication/logical/conflict.c | 308 ++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 37 +++
src/bin/pg_dump/t/002_pg_dump.pl | 6 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 55 ++++
src/include/nodes/parsenodes.h | 3 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 72 ++++
src/test/regress/sql/subscription.sql | 32 ++
17 files changed, 851 insertions(+), 87 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index bee7e02983..786a99dbfc 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1571,7 +1571,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1585,85 +1585,14 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered in the following <firstterm>conflict</firstterm>
- cases:
- <variablelist>
- <varlistentry>
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_differ</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_differ</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ Additional logging is triggered for specific <literal>conflict_resolvers</literal>.
+ Users can also configure <literal>conflict_types</literal> while creating
+ the subscription.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for details on conflict_types and conflict_resolvers.
+
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the log.
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d007..e7b39f2000
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
</synopsis>
</refsynopsisdiv>
@@ -345,6 +346,17 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d9421..cacce6979e
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -25,6 +25,7 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,172 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-differ">
+ <term><literal>update_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-differ">
+ <term><literal>delete_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index b925c464ae..c8cea08899 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -28,6 +29,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +39,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -112,6 +115,7 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub,
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
+static void check_conflict_detection(void);
/*
@@ -439,6 +443,32 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
}
+/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+
+ if (!stmtresolvers)
+ return;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+
+ /* validate the conflict type and resolver */
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* update the corresponding resolver for the given conflict type */
+ resolvers[type].resolver = defGetString(defel);
+ }
+}
+
/*
* Add publication names from the list to a string.
*/
@@ -583,6 +613,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
/*
* Parse and check options.
@@ -597,6 +628,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /*
+ * Parse and check conflict resolvers. Initialize with default values
+ */
+ if (stmt->resolvers)
+ check_conflict_detection();
+
+ SetDefaultResolvers(conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -726,6 +766,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
/*
* Connect to remote side to execute requested commands and fetch table
* info.
@@ -1581,6 +1624,22 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ check_conflict_detection();
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1891,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
@@ -2536,3 +2598,18 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be detected, "
+ "and the origin and commit timestamp for the local row will not be logged."));
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 84cef57a70..f54163f3cb 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -769,7 +769,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8754,6 +8754,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10688,7 +10693,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10696,6 +10701,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10802,6 +10808,17 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+
;
/*****************************************************************************
@@ -17725,6 +17742,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18353,6 +18371,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 0bc7959980..ad79f816dd 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,12 +15,26 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -31,6 +45,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_DIFFER] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_DIFFER] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_DIFFER] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_DIFFER] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -486,3 +549,248 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *CTR = NULL;
+ List *res = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ CTR = palloc(sizeof(ConflictTypeResolver));
+ CTR->conflict_type = defel->defname;
+ CTR->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+
+ res = lappend(res, CTR);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolver for a conflict type in
+ * pg_subscription_conflict system catalog
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int type;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ for (type = 0; type < resolvers_cnt; type++)
+ {
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[type].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[type].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index b6e01d3d29..73e90cb6c1 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5254,6 +5254,43 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ PQExpBuffer InQry = createPQExpBuffer();
+ PGresult *res;
+ int i_confrtype;
+ int i_confrres;
+
+ /* get the conflict types and their resolvers from the catalog */
+ appendPQExpBuffer(InQry,
+ "SELECT confrtype, confrres "
+ "FROM pg_catalog.pg_subscription_conflict"
+ " WHERE confsubid = %u;\n", subinfo->dobj.catId.oid);
+ res = ExecuteSqlQuery(fout, InQry->data, PGRES_TUPLES_OK);
+
+ i_confrtype = PQfnumber(res, "confrtype");
+ i_confrres = PQfnumber(res, "confrres");
+
+ if (PQntuples(res) > 0)
+ {
+ appendPQExpBufferStr(query, ") CONFLICT RESOLVER (");
+
+ for (i = 0; i < PQntuples(res); ++i)
+ {
+ if (i == 0)
+ appendPQExpBuffer(query, "%s = '%s'",
+ PQgetvalue(res, i, i_confrtype),
+ PQgetvalue(res, i, i_confrres));
+ else
+ appendPQExpBuffer(query, ", %s = '%s'",
+ PQgetvalue(res, i, i_confrtype),
+ PQgetvalue(res, i, i_confrres));
+
+ }
+ }
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 5bcc2244d5..2fbcac0a7a 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2956,7 +2956,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub1 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub1');\E
+ \QCREATE SUBSCRIPTION sub1 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub1') CONFLICT RESOLVER (insert_exists = 'error', update_differ = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_differ = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2967,7 +2967,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false, origin = none);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub2 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub2', origin = none);\E
+ \QCREATE SUBSCRIPTION sub2 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub2', origin = none) CONFLICT RESOLVER (insert_exists = 'error', update_differ = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_differ = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2978,7 +2978,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false, origin = any);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3');\E
+ \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3') CONFLICT RESOLVER (insert_exists = 'error', update_differ = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_differ = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..c8b37c2bef
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTSUBOID, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 124d853e49..f11e794627 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4188,6 +4188,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4200,6 +4201,7 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4210,6 +4212,7 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f8659078ce..d661a06b36 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -375,6 +375,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 02cb84da7e..fac3f8335e 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -42,6 +42,47 @@ typedef enum
*/
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -54,5 +95,13 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictBySubid(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..997f8171e6 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -399,6 +399,78 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | apply_remote
+ delete_missing | skip
+ insert_exists | error
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | keep_local
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_differ = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+----------------+--------------
+ delete_differ | keep_local
+ delete_missing | skip
+ insert_exists | apply_remote
+ update_differ | apply_remote
+ update_exists | error
+ update_missing | skip
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..d21b75ff69 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,38 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_differ = 'keep_local' );
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_differ = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
--
2.34.1
v10-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v10-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From b8f565ac71e788b73a4a756fbeb905922291ac7e Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 22 Aug 2024 15:51:06 +0530
Subject: [PATCH v10 2/3] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_differ: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_differ: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 68 +-
src/backend/replication/logical/conflict.c | 222 +++++--
src/backend/replication/logical/worker.c | 369 ++++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 13 +-
src/test/subscription/meson.build | 1 +
.../subscription/t/034_conflict_resolver.pl | 581 ++++++++++++++++++
7 files changed, 1114 insertions(+), 145 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc962..4059707a27 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,13 +550,58 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
+/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
+ TupleTableSlot **conflictslot, ConflictType type,
+ ConflictResolver resolver, TupleTableSlot *slot,
+ Oid subid, bool apply_remote)
+{
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -565,7 +610,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -588,6 +634,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -601,6 +649,20 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index ad79f816dd..f7f54a7933 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -31,6 +31,7 @@
#include "replication/conflict.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
@@ -98,12 +99,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -146,8 +149,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -166,26 +169,36 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -253,17 +266,25 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -274,13 +295,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -290,47 +312,62 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_DIFFER:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"),
+ applymsg);
break;
case CT_DELETE_DIFFER:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -634,6 +671,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
@@ -794,3 +894,47 @@ RemoveSubscriptionConflictBySubid(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 38c2895307..be23500ffe 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2704,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2727,47 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_DIFFER,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_DIFFER, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2777,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2911,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2928,51 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_DIFFER,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_DIFFER,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3020,19 +3105,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3058,6 +3145,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3070,47 +3160,84 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_DIFFER,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_DIFFER,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_DIFFER,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_differ is set to skip. Ignore remote update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3122,23 +3249,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_differ conflict is
+ * detected for the found tuple and the resolver is in
+ * favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3179,12 +3335,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3200,22 +3364,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00..f3909759f8 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index fac3f8335e..4b97003e2b 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -88,12 +90,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -103,5 +107,10 @@ extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..3be8b52d8d
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,581 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_differ|apply_remote
+delete_missing|skip
+insert_exists|error
+update_differ|apply_remote
+update_exists|error
+update_missing|skip),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#########################################
+# Test 'apply_remote' for 'delete_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_differ/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+#########################################
+# Test 'keep_local' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_differ/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'delete_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of delete_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_differ./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#########################################
+# Test 'apply_remote' for 'update_differ'
+#########################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_differ/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+#########################################
+# Test 'keep_local' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_differ/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+#########################################
+# Test 'error' for 'update_differ'
+#########################################
+
+# Change CONFLICT RESOLVER of update_differ to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_differ = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_differ./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v10-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v10-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From ee604d3c0ed1c49057650731dcb06a23a663536d Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Mon, 26 Aug 2024 12:29:28 +0530
Subject: [PATCH v10 3/3] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 44 ++++++--
src/backend/replication/logical/worker.c | 84 ++++++++++++---
src/include/executor/executor.h | 3 +-
.../subscription/t/034_conflict_resolver.pl | 100 ++++++++++++++++++
4 files changed, 209 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4059707a27..f944df761c 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -566,7 +566,7 @@ static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote)
+ Oid subid, bool apply_remote, ItemPointer tupleid)
{
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
@@ -579,7 +579,7 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
* otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -636,6 +636,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool conflict = false;
ConflictResolver resolver;
bool apply_remote = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -656,11 +657,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
NULL, subid);
- /* Check for conflict and return to caller for resolution if found */
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'insert_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
if (resolver != CR_ERROR &&
has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote))
+ apply_remote, &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -717,7 +723,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -743,6 +750,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -756,6 +765,25 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'update_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, resolver, slot, subid,
+ apply_remote, tid))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index be23500ffe..bd44d5b352 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2517,8 +2518,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2670,7 +2672,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2695,14 +2698,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2710,10 +2712,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2754,6 +2757,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2766,7 +2771,32 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3263,6 +3293,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* detected for the found tuple and the resolver is in
* favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3270,7 +3302,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f3909759f8..291d5dde36 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 3be8b52d8d..024b212588 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
--
2.34.1
On Mon, Aug 26, 2024 at 9:05 AM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1]. The detect_conflict option has
been removed, and conflict detection is now enabled by default. This
change required the following updates in resolver patches:
patch-0001:
- Removed dependency on the detect_conflict option. Now, default
conflict resolvers are set on CREATE SUBSCRIPTION if no values are
provided.
- To keep the behavior unchanged, the default resolvers are now set as -
insert_exists = error
update_exists = error
update_differ = apply_remote
update_missing = skip
delete_missing = skip
delete_differ = apply_remote
- Added documentation for conflict resolvers.patch-0002:
- Removed dependency on the detect_conflict option.
- Updated test cases in 034_conflict_resolver.pl to reflect new
default resolvers and the removal of the detect_conflict option.patch-0003:
- Implemented resolver for the update_exists conflict type. Supported
resolvers are: apply_remote, keep_local, error.Thanks Nisha for the patches, I was running some tests on
update_exists and found this case wherein it misses to LOG one
conflict out of 3.create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);Pub: update tab set a=2,b=40,c=70 where a=1;
Here it logs update_exists conflict and the resolution for Key
(b)=(40) and Key (c)=(70) but misses to LOG first one which is with
Key (a)=(2).
Fixed.
Thanks,
Nisha
On Mon, Aug 26, 2024 at 2:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Aug 22, 2024 at 3:45 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?It makes sense to have a RESET on the lines of (a) and (b). At this
stage, we should do minimal in extending the syntax. How about RESET
CONFLICT RESOLVER ALL for (a)?~~
3) Regarding update_exists:
3a)
Currently update_exists resolver patch is kept separate. The reason
being, it performs resolution which will need deletion of multiple
rows. It will be good to discuss if we want to target this in the
first draft. Please see the example:create table tab (a int primary key, b int unique, c int unique);
Pub: insert into tab values (1,1,1);
Sub:
insert into tab values (2,20,30);
insert into tab values (3,40,50);
insert into tab values (4,60,70);Pub: update tab set a=2,b=40,c=70 where a=1;
The above 'update' on pub will result in 'update_exists' on sub and if
resolution is in favour of 'apply', then it will conflict with all the
three local rows of subscriber due to unique constraint present on all
three columns. Thus in order to resolve the conflict, it will have to
delete these 3 rows on sub:2,20,30
3,40,50
4,60,70
and then update 1,1,1 to 2,40,70.Just need opinion on if we shall target this in the initial draft.
This case looks a bit complicated. It seems there is no other
alternative than to delete the multiple rows. It is better to create a
separate top-up patch for this and we can discuss in detail about this
once the basic patch is in better shape.
v9 onwards the patch-0003 is a separate top-up patch implementing update_exists.
3b)
If we plan to implement this, we need to work on optimal design where
we can find all the conflicting rows at once and delete those.
Currently the implementation has been done using recursion i.e. find
one conflicting row, then delete it and then next and so on i.e. we
call apply_handle_update_internal() recursively. On initial code
review, I feel it is doable to scan all indexes at once and get
conflicting-tuple-ids in one go and get rid of recursion. It can be
attempted once we decide on 3a.I suggest following the simplest strategy (even if that means calling
the update function recursively) by adding comments on the optimal
strategy. We can optimize it later as well.~~
4)
Now for insert_exists and update_exists, we are doing a pre-scan of
all unique indexes to find conflict. Also there is post-scan to figure
out if the conflicting row is inserted meanwhile. This needs to be
reviewed for optimization. We need to avoid pre-scan wherever
possible. I think the only case for which it can be avoided is
'ERROR'. For the cases where resolver is in favor of remote-apply, we
need to check conflict beforehand to avoid rollback of already
inserted data. And for the case where resolver is in favor of skipping
the change, then too we should know beforehand about the conflict to
avoid heap-insertion and rollback. Thoughts?It makes sense to skip the pre-scan wherever possible. Your analysis
sounds reasonable to me.
Done.
--
Thanks,
Nisha
On Fri, Aug 23, 2024 at 10:39 AM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Aug 22, 2024 at 3:44 PM shveta malik <shveta.malik@gmail.com> wrote:
For clock-skew and timestamp based resolution, if needed, I will post
another email for the design items where suggestions are needed.Please find issues which need some thoughts and approval for
time-based resolution and clock-skew.1)
Time based conflict resolution and two phase transactions:Time based conflict resolution (last_update_wins) is the one
resolution which will not result in data-divergence considering
clock-skew is taken care of. But when it comes to two-phase
transactions, it might not be the case. For two-phase transaction, we
do not have commit timestamp when the changes are being applied. Thus
for time-based comparison, initially it was decided to user prepare
timestamp but it may result in data-divergence. Please see the
example at [1].Example at [1] is a tricky situation, and thus in the initial draft,
we decided to restrict usage of 2pc and CDR together. The plan is:a) During Create subscription, if the user has given last_update_wins
resolver for any conflict_type and 'two_phase' is also enabled, we
ERROR out.
b) During Alter subscription, if the user tries to update resolver to
'last_update_wins' but 'two_phase' is enabled, we error out.Another solution could be to save both prepare_ts and commit_ts. And
when any txn comes for conflict resolution, we first check if
prepare_ts is available, use that else use commit_ts. Availability of
prepare_ts would indicate it was a prepared txn and thus even if it is
committed, we should use prepare_ts for comparison for consistency.
This will have some overhead of storing prepare_ts along with
commit_ts. But if the number of prepared txns are reasonably small,
this overhead should be less.We currently plan to go with restricting 2pc and last_update_wins
together, unless others have different opinions.~~
2)
parallel apply worker and conflict-resolution:
As discussed in [2] (see last paragraph in [2]), for streaming of
in-progress transactions by parallel worker, we do not have
commit-timestamp with each change and thus it makes sense to disable
parallel apply worker with CDR. The plan is to not start parallel
apply worker if 'last_update_wins' is configured for any
conflict_type.~~
3)
parallel apply worker and clock skew management:
Regarding clock-skew management as discussed in [3], we will wait for
the local clock to come within tolerable range during 'begin' rather
than before 'commit'. And this wait needs commit-timestamp in the
beginning, thus we plan to restrict starting pa-worker even when
clock-skew related GUCs are configured.Earlier we had restricted both 2pc and parallel worker worker start
when detect_conflict was enabled, but now since detect_conflict
parameter is removed, we will change the implementation to restrict
all 3 above cases when last_update_wins is configured. When the
changes are done, we will post the patch.~~
4)
<not related to timestamp and clock skew>
Earlier when 'detect_conflict' was enabled, we were giving WARNING if
'track_commit_timestamp' was not enabled. This was during CREATE and
ALTER subscription. Now with this parameter removed, this WARNING has
also been removed. But I think we need to bring back this WARNING.
Currently default resolvers set may work without
'track_commit_timestamp' but when user gives CONFLICT RESOLVER in
create-sub or alter-sub explicitly making them configured to
non-default values (or say any values, does not matter if few are
defaults), we may still emit this warning to alert user:2024-07-26 09:14:03.152 IST [195415] WARNING: conflict detection
could be incomplete due to disabled track_commit_timestamp
2024-07-26 09:14:03.152 IST [195415] DETAIL: Conflicts update_differ
and delete_differ cannot be detected, and the origin and commit
timestamp for the local row will not be logged.Thoughts?
If we emit this WARNING during each resolution, then it may flood our
log files, thus it seems better to emit it during create or alter
subscription instead of during resolution.
Done.
v10 has implemented the suggested warning when a user gives CONFLICT
RESOLVER in create-sub or alter-sub explicitly.
Thanks,
Nisha
On Tue, Aug 27, 2024 at 1:51 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Please find v10 patch-set. Changes are:
1) patch-001:
- Corrected a patch application warning.
- Added support for pg_dump.
- As suggested in pt.4 of [1]: added a warning during CREATE and
ALTER subscription when track_commit_timestamp is OFF.2) patch-002 & patch-003:
- Reduced code duplication in execReplication.c
- As suggested in pt.4 of [2]: Optimized the pre-scan for
insert_exists and update_exists cases when resolver is set to ERROR.
- Fixed a bug reported by Shveta in [3]Thank You Ajin for working on pg_dump support changes.
Thank You for the patches. Few comments for pg_dump
1)
If there are multiple subscriptions with different resolver
configuration, pg_dump currently dumps resolver in different orders
for each subscription. It is not a problem, but it will be better to
have it in the same order. We can have an order-by in the pg_dump's
code while querying resolvers.
2)
Currently pg_dump is dumping even the default resolvers configuration.
As an example if I have not changed default configuration for say
sub1, it still dumps all:
CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 WITH (....)
CONFLICT RESOLVER (insert_exists = 'error', update_differ =
'apply_remote', update_exists = 'error', update_missing = 'skip',
delete_differ = 'apply_remote', delete_missing = 'skip');
I am not sure if we need to dump default resolvers. Would like to know
what others think on this.
3)
Why in 002_pg_dump.pl we have default resolvers set explicitly?
thanks
Shveta
On Wed, Aug 28, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
2)
Currently pg_dump is dumping even the default resolvers configuration.
As an example if I have not changed default configuration for say
sub1, it still dumps all:CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 WITH (....)
CONFLICT RESOLVER (insert_exists = 'error', update_differ =
'apply_remote', update_exists = 'error', update_missing = 'skip',
delete_differ = 'apply_remote', delete_missing = 'skip');I am not sure if we need to dump default resolvers. Would like to know
what others think on this.3)
Why in 002_pg_dump.pl we have default resolvers set explicitly?In 003_pg_dump.pl, default resolvers are not set explicitly, that is the
regexp to check the pg_dump generated command for creating subscriptions.
This is again connected to your 2nd question.
regards,
Ajin Cherian
Fujitsu Australia
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
2)
Currently pg_dump is dumping even the default resolvers configuration.
As an example if I have not changed default configuration for say
sub1, it still dumps all:CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 WITH (....)
CONFLICT RESOLVER (insert_exists = 'error', update_differ =
'apply_remote', update_exists = 'error', update_missing = 'skip',
delete_differ = 'apply_remote', delete_missing = 'skip');I am not sure if we need to dump default resolvers. Would like to know
what others think on this.3)
Why in 002_pg_dump.pl we have default resolvers set explicitly?In 003_pg_dump.pl, default resolvers are not set explicitly, that is the regexp to check the pg_dump generated command for creating subscriptions. This is again connected to your 2nd question.
Okay so we may not need this change if we plan to *not *dump defaults
in pg_dump.
Another point about 'defaults' is regarding insertion into the
pg_subscription_conflict table. We currently do insert default
resolvers into 'pg_subscription_conflict' even if the user has not
explicitly configured them. I think it is okay to insert defaults
there as the user will be able to know which resolver is picked for
any conflict type. But again, I would like to know the thoughts of
others on this.
thanks
Shveta
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
The review is WIP. Please find a few comments on patch001.
1)
logical-repliction.sgmlL
+ Additional logging is triggered for specific conflict_resolvers.
Users can also configure conflict_types while creating the
subscription. Refer to section CONFLICT RESOLVERS for details on
conflict_types and conflict_resolvers.
Can we please change it to:
Additional logging is triggered in various conflict scenarios, each
identified as a conflict type. Users have the option to configure a
conflict resolver for each conflict type when creating a subscription.
For more information on the conflict types detected and the supported
conflict resolvers, refer to the section <CONFLICT RESOLVERS>
2)
SetSubConflictResolver
+ for (type = 0; type < resolvers_cnt; type++)
'type' does not look like the correct name here. The variable does not
state conflict_type, it is instead a resolver-array-index, so please
rename accordingly. Maybe idx or res_idx?
3)
CreateSubscription():
+ if (stmt->resolvers)
+ check_conflict_detection();
3a) We can have a comment saying warn users if prerequisites are not met.
3b) Also, I do not find the name 'check_conflict_detection'
appropriate. One suggestion could be
'conf_detection_check_prerequisites' (similar to
replorigin_check_prerequisites)
3c) We can move the below comment after check_conflict_detection() as
it makes more sense there.
/*
* Parse and check conflict resolvers. Initialize with default values
*/
4)
Should we allow repetition/duplicates of 'conflict_type=..' in CREATE
and ALTER SUB? As an example:
ALTER SUBSCRIPTION sub1 CONFLICT RESOLVER (insert_exists =
'apply_remote', insert_exists = 'error');
Such a repetition works for Create-Sub but gives some internal error
for alter-sub. (ERROR: tuple already updated by self). Behaviour
should be the same for both. And if we give an error, it should be
some user understandable one. But I would like to know the opinions of
others. Shall it give an error or the last one should be accepted as
valid configuration in case of repetition?
5)
GetAndValidateSubsConflictResolverList():
+ ConflictTypeResolver *CTR = NULL;
We can change the name to a more appropriate one similar to other
variables. It need not be in all capital.
thanks
Shveta
On Wed, Aug 28, 2024 at 4:07 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
The review is WIP. Please find a few comments on patch001.
More comments on ptach001 in continuation of previous comments:
6)
SetDefaultResolvers() can be called from
parse_subscription_conflict_resolvers() itself. This will be similar
to how parse_subscription_options() sets defaults internally.
7)
parse_subscription_conflict_resolvers():
+ if (!stmtresolvers)
+ return;
I think we do not need the above, 'foreach' will take care of it.
Since we do not have any logic after foreach, we should be good
without the above check explicitly added.
8)
I think SetSubConflictResolver() should be moved before
replorigin_create(). We can insert resolver entries immediately after
we insert subscription entries.
9)
check_conflict_detection/conf_detection_check_prerequisites shall be
moved to conflict.c file.
10)
validate_conflict_type_and_resolver():
Please mention in header that:
It returns an enum ConflictType corresponding to the conflict type
string passed by the caller.
11)
UpdateSubConflictResolvers():
11a) Rename CTR similar to other variables.
11b) Please correct the header as we deal with multiple conflict-types
in it instead of 1.
Suggestion: Update the subscription's conflict resolvers in
pg_subscription_conflict system catalog for the given conflict types.
12)
SetSubConflictResolver():
12a) I think we do not need 'replaces' during INSERT and thus this is
not needed:
+ memset(replaces, false, sizeof(replaces));
12b)
Shouldn't below be outside of loop:
+ memset(nulls, false, sizeof(nulls));
13)
Shall we rename RemoveSubscriptionConflictBySubid with
RemoveSubscriptionConflictResolvers()? 'BySubid' is not needed as we
have Subscription in the name and we do not have any other variation
of removal.
14)
We shall rename pg_subscription_conflict_sub_index to
pg_subscription_conflict_confsubid_confrtype_index to give more
clarity that it is any index on subid and conftype
And SubscriptionConflictSubIndexId to SubscriptionConflictSubidTypeIndexId
And SUBSCRIPTIONCONFLICTSUBOID to SUBSCRIPTIONCONFLMAP
15)
conflict.h:
+ See ConflictTypeResolverMap in conflcit.c to find out which all
conflcit.c --> conflict.c
16)
subscription.sql:
16a) add one more test case for 'fail' scenario where both conflict
type and resolver are valid but resolver is not for that particular
conflict type.
16b)
--try setting resolvers for few types
Change to below (similar to other comments)
-- ok - valid conflict types and resolvers
16c)
-- ok - valid conflict type and resolver
maybe change to: -- ok - valid conflict types and resolvers
thanks
Shveta
On Fri, Aug 23, 2024 at 10:39 AM shveta malik <shveta.malik@gmail.com> wrote:
Please find issues which need some thoughts and approval for
time-based resolution and clock-skew.1)
Time based conflict resolution and two phase transactions:Time based conflict resolution (last_update_wins) is the one
resolution which will not result in data-divergence considering
clock-skew is taken care of. But when it comes to two-phase
transactions, it might not be the case. For two-phase transaction, we
do not have commit timestamp when the changes are being applied. Thus
for time-based comparison, initially it was decided to user prepare
timestamp but it may result in data-divergence. Please see the
example at [1].Example at [1] is a tricky situation, and thus in the initial draft,
we decided to restrict usage of 2pc and CDR together. The plan is:a) During Create subscription, if the user has given last_update_wins
resolver for any conflict_type and 'two_phase' is also enabled, we
ERROR out.
b) During Alter subscription, if the user tries to update resolver to
'last_update_wins' but 'two_phase' is enabled, we error out.Another solution could be to save both prepare_ts and commit_ts. And
when any txn comes for conflict resolution, we first check if
prepare_ts is available, use that else use commit_ts. Availability of
prepare_ts would indicate it was a prepared txn and thus even if it is
committed, we should use prepare_ts for comparison for consistency.
This will have some overhead of storing prepare_ts along with
commit_ts. But if the number of prepared txns are reasonably small,
this overhead should be less.
Yet another idea is that if the conflict is detected and the
resolution strategy is last_update_wins then from that point we start
writing all the changes to the file similar to what we do for
streaming mode and only once commit_prepared arrives, we will read and
apply changes. That will solve this problem.
We currently plan to go with restricting 2pc and last_update_wins
together, unless others have different opinions.
Sounds reasonable but we should add comments on the possible solution
like the one I have mentioned so that we can extend it afterwards.
~~
2)
parallel apply worker and conflict-resolution:
As discussed in [2] (see last paragraph in [2]), for streaming of
in-progress transactions by parallel worker, we do not have
commit-timestamp with each change and thus it makes sense to disable
parallel apply worker with CDR. The plan is to not start parallel
apply worker if 'last_update_wins' is configured for any
conflict_type.
The other idea is that we can let the changes written to file if any
conflict is detected and then at commit time let the remaining changes
be applied by apply worker. This can introduce some complexity, so
similar to two_pc we can extend this functionality later.
~~
3)
parallel apply worker and clock skew management:
Regarding clock-skew management as discussed in [3], we will wait for
the local clock to come within tolerable range during 'begin' rather
than before 'commit'. And this wait needs commit-timestamp in the
beginning, thus we plan to restrict starting pa-worker even when
clock-skew related GUCs are configured.Earlier we had restricted both 2pc and parallel worker worker start
when detect_conflict was enabled, but now since detect_conflict
parameter is removed, we will change the implementation to restrict
all 3 above cases when last_update_wins is configured. When the
changes are done, we will post the patch.
At this stage, we are not sure how we want to deal with clock skew.
There is an argument that clock-skew should be handled outside the
database, so we can probably have the clock-skew-related stuff in a
separate patch.
~~
4)
<not related to timestamp and clock skew>
Earlier when 'detect_conflict' was enabled, we were giving WARNING if
'track_commit_timestamp' was not enabled. This was during CREATE and
ALTER subscription. Now with this parameter removed, this WARNING has
also been removed. But I think we need to bring back this WARNING.
Currently default resolvers set may work without
'track_commit_timestamp' but when user gives CONFLICT RESOLVER in
create-sub or alter-sub explicitly making them configured to
non-default values (or say any values, does not matter if few are
defaults), we may still emit this warning to alert user:2024-07-26 09:14:03.152 IST [195415] WARNING: conflict detection
could be incomplete due to disabled track_commit_timestamp
2024-07-26 09:14:03.152 IST [195415] DETAIL: Conflicts update_differ
and delete_differ cannot be detected, and the origin and commit
timestamp for the local row will not be logged.Thoughts?
If we emit this WARNING during each resolution, then it may flood our
log files, thus it seems better to emit it during create or alter
subscription instead of during resolution.
Sounds reasonable.
--
With Regards,
Amit Kapila.
Here is the v11 patch-set. Changes are:
1) Updated conflict type names in accordance with the recent commit[1]/messages/by-id/CAA4eK1+poV1dDqK=hdv-Zh2m2kBdB=s1TBk8MwscgdzULBonbw@mail.gmail.com as -
update_differ --> update_origin_differs
delete_differ --> delete_origin_differs
2) patch-001:
- Implemented the RESET command to restore the default resolvers as
suggested in pt.2a & 2b in [2]/messages/by-id/CAJpy0uBrXZE6LLofX5tc8WOm5F+FNgnQjRLQerOY8cOqqvtrNg@mail.gmail.com
3) patch-004:
- Rebased the patch which implements last_update_wins resolver and
clock-skew management.
- Restricts the setting of two_phase and last_update_wins together
- Prevents the start of parallel apply-worker if 'last_update_wins'
is configured for any conflict_type.
- Added test cases for last_update_wins in 034_conflict_resolver.pl
Thanks, Shveta for your help in patch-004 solutions and thank you Ajin
for providing RESET command changes(patch-001).
[1]: /messages/by-id/CAA4eK1+poV1dDqK=hdv-Zh2m2kBdB=s1TBk8MwscgdzULBonbw@mail.gmail.com
[2]: /messages/by-id/CAJpy0uBrXZE6LLofX5tc8WOm5F+FNgnQjRLQerOY8cOqqvtrNg@mail.gmail.com
--
Thanks,
Nisha
Attachments:
v11-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v11-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From 3c035bf58bd9094417dc83d1bb450fae3f3aa2db Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 29 Aug 2024 16:26:38 +0530
Subject: [PATCH v11 1/4] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for reseting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 89 +---
doc/src/sgml/ref/alter_subscription.sgml | 12 +
doc/src/sgml/ref/create_subscription.sgml | 167 ++++++++
src/backend/commands/subscriptioncmds.c | 101 +++++
src/backend/parser/gram.y | 48 ++-
src/backend/replication/logical/conflict.c | 384 ++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 37 ++
src/bin/pg_dump/t/002_pg_dump.pl | 6 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 55 +++
src/include/nodes/parsenodes.h | 6 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 50 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 101 +++++
src/test/regress/sql/subscription.sql | 43 ++
17 files changed, 1018 insertions(+), 87 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 46917f9f94..37145d21a8 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,85 +1582,14 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered in the following <firstterm>conflict</firstterm>
- cases:
- <variablelist>
- <varlistentry>
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry>
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ Additional logging is triggered for specific <literal>conflict_resolvers</literal>.
+ Users can also configure <literal>conflict_types</literal> while creating
+ the subscription.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for details on conflict_types and conflict_resolvers.
+
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the log.
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d007..e7b39f2000
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
</synopsis>
</refsynopsisdiv>
@@ -345,6 +346,17 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d9421..cacce6979e
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -25,6 +25,7 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,172 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-differ">
+ <term><literal>update_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-differ">
+ <term><literal>delete_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc636b8..38dd67487e 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -28,6 +29,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +39,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -112,6 +115,7 @@ static List *merge_publications(List *oldpublist, List *newpublist, bool addpub,
static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname, char *err);
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
+static void check_conflict_detection(void);
/*
@@ -439,6 +443,32 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
}
+/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+
+ if (!stmtresolvers)
+ return;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+
+ /* validate the conflict type and resolver */
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* update the corresponding resolver for the given conflict type */
+ resolvers[type].resolver = defGetString(defel);
+ }
+}
+
/*
* Add publication names from the list to a string.
*/
@@ -583,6 +613,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
/*
* Parse and check options.
@@ -597,6 +628,15 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /*
+ * Parse and check conflict resolvers. Initialize with default values
+ */
+ if (stmt->resolvers)
+ check_conflict_detection();
+
+ SetDefaultResolvers(conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -726,6 +766,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
/*
* Connect to remote side to execute requested commands and fetch table
* info.
@@ -1581,6 +1624,46 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ check_conflict_detection();
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
+ {
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+
+ /* remove existing conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
+ /*
+ * create list of conflict resolvers and set them in the
+ * catalog
+ */
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER:
+ {
+ /*
+ * reset the conflict resolver for this conflict type to its
+ * default
+ */
+ ResetConflictResolver(subid, stmt->conflict_type);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1915,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictBySubid(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
@@ -2536,3 +2622,18 @@ defGetStreamingMode(DefElem *def)
def->defname)));
return LOGICALREP_STREAM_OFF; /* keep compiler quiet */
}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be detected, "
+ "and the origin and commit timestamp for the local row will not be logged."));
+}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 84cef57a70..be8748788b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -612,6 +612,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -769,7 +770,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8754,6 +8755,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10688,7 +10694,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10696,6 +10702,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10802,6 +10809,39 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
;
/*****************************************************************************
@@ -17725,6 +17765,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18353,6 +18394,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index a1437d4f77..2ddb2ff59e 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,12 +15,26 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -31,6 +45,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -486,3 +549,324 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *CTR = NULL;
+ List *res = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ CTR = palloc(sizeof(ConflictTypeResolver));
+ CTR->conflict_type = defel->defname;
+ CTR->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+
+ res = lappend(res, CTR);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolver for a conflict type in
+ * pg_subscription_conflict system catalog
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver conflictResolver;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ Relation pg_subscription_conflict;
+ HeapTuple oldtup,
+ newtup;
+ bool valid = false;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Get the index for this conflict_type */
+ for (idx = CT_MIN; idx <= CT_MAX; idx++)
+ {
+ if (strcmp(ConflictTypeNames[idx], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* get the default resolver for this conflict_type */
+ conflictResolver.resolver = ConflictResolverNames[ConflictTypeDefaultResolvers[idx]];
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u", conflict_type, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ oldtup, Anum_pg_subscription_conflict_confrres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /* check if current resolver is the default one, if not update it */
+ if (strcmp(cur_conflict_res, conflictResolver.resolver) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conflictResolver.resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int type;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ for (type = 0; type < resolvers_cnt; type++)
+ {
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[type].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[type].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index f7720ad53b..1491f5e0f7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -5254,6 +5254,43 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ PQExpBuffer InQry = createPQExpBuffer();
+ PGresult *res;
+ int i_confrtype;
+ int i_confrres;
+
+ /* get the conflict types and their resolvers from the catalog */
+ appendPQExpBuffer(InQry,
+ "SELECT confrtype, confrres "
+ "FROM pg_catalog.pg_subscription_conflict"
+ " WHERE confsubid = %u;\n", subinfo->dobj.catId.oid);
+ res = ExecuteSqlQuery(fout, InQry->data, PGRES_TUPLES_OK);
+
+ i_confrtype = PQfnumber(res, "confrtype");
+ i_confrres = PQfnumber(res, "confrres");
+
+ if (PQntuples(res) > 0)
+ {
+ appendPQExpBufferStr(query, ") CONFLICT RESOLVER (");
+
+ for (i = 0; i < PQntuples(res); ++i)
+ {
+ if (i == 0)
+ appendPQExpBuffer(query, "%s = '%s'",
+ PQgetvalue(res, i, i_confrtype),
+ PQgetvalue(res, i, i_confrres));
+ else
+ appendPQExpBuffer(query, ", %s = '%s'",
+ PQgetvalue(res, i, i_confrtype),
+ PQgetvalue(res, i, i_confrres));
+
+ }
+ }
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index 5bcc2244d5..8e2522d601 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -2956,7 +2956,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub1 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub1');\E
+ \QCREATE SUBSCRIPTION sub1 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub1') CONFLICT RESOLVER (insert_exists = 'error', update_origin_differs = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_origin_differs = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2967,7 +2967,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false, origin = none);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub2 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub2', origin = none);\E
+ \QCREATE SUBSCRIPTION sub2 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub2', origin = none) CONFLICT RESOLVER (insert_exists = 'error', update_origin_differs = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_origin_differs = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
@@ -2978,7 +2978,7 @@ my %tests = (
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
WITH (connect = false, origin = any);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3');\E
+ \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3') CONFLICT RESOLVER (insert_exists = 'error', update_origin_differs = 'apply_remote', update_exists = 'error', update_missing = 'skip', delete_origin_differs = 'apply_remote', delete_missing = 'skip');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..c8b37c2bef
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLICTSUBOID, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 124d853e49..bcb9a90de7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4188,6 +4188,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4200,6 +4201,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4210,6 +4214,8 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f8659078ce..d661a06b36 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -375,6 +375,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index ca797fb41c..8d5799e4ce 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -42,6 +42,47 @@ typedef enum
*/
} ConflictType;
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -54,5 +95,14 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictBySubid(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void ResetConflictResolver(Oid subid, char *conflict_type);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..eea6b6aff2 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -399,6 +399,107 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | keep_local
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..f709b03ad5 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,49 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+--try setting resolvers for few types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- ok - valid conflict type and resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
--
2.34.1
v11-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v11-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 9e2ae3e9b9e1c1ed55ee5a480b8dea942f7dcdc1 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 29 Aug 2024 17:35:07 +0530
Subject: [PATCH v11 2/4] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 68 +-
src/backend/replication/logical/conflict.c | 221 +++++--
src/backend/replication/logical/worker.c | 370 ++++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 14 +-
src/test/subscription/meson.build | 1 +
.../subscription/t/034_conflict_resolver.pl | 581 ++++++++++++++++++
7 files changed, 1114 insertions(+), 146 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc962..4059707a27 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,13 +550,58 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
+/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
+ TupleTableSlot **conflictslot, ConflictType type,
+ ConflictResolver resolver, TupleTableSlot *slot,
+ Oid subid, bool apply_remote)
+{
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -565,7 +610,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -588,6 +634,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -601,6 +649,20 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 2ddb2ff59e..9fd96d4979 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -31,6 +31,7 @@
#include "replication/conflict.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
+#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
@@ -98,12 +99,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -146,8 +149,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -166,26 +169,36 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
+
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -253,17 +266,24 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -274,13 +294,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -290,47 +311,62 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"),
+ applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -634,6 +670,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLICTSUBOID,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLICTSUBOID,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
@@ -870,3 +969,47 @@ RemoveSubscriptionConflictBySubid(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 0fb577d328..c773a7016b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2704,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2727,47 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2777,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2911,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2928,51 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3020,19 +3105,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3058,6 +3145,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3070,47 +3160,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3122,23 +3250,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3179,12 +3336,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3200,22 +3365,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00..f3909759f8 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 8d5799e4ce..c0b7b91063 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -88,12 +90,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -104,5 +108,9 @@ extern ConflictType validate_conflict_type_and_resolver(const char *conflict_typ
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
-
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..784f89fe2d
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,581 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v11-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v11-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From 66940ed045e7dc4ca85ce31798eac29341af2230 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 29 Aug 2024 17:38:00 +0530
Subject: [PATCH v11 3/4] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 44 ++++++--
src/backend/replication/logical/worker.c | 84 ++++++++++++---
src/include/executor/executor.h | 3 +-
.../subscription/t/034_conflict_resolver.pl | 100 ++++++++++++++++++
4 files changed, 209 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4059707a27..f944df761c 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -566,7 +566,7 @@ static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote)
+ Oid subid, bool apply_remote, ItemPointer tupleid)
{
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
@@ -579,7 +579,7 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
* otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -636,6 +636,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool conflict = false;
ConflictResolver resolver;
bool apply_remote = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -656,11 +657,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
NULL, subid);
- /* Check for conflict and return to caller for resolution if found */
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'insert_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
if (resolver != CR_ERROR &&
has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote))
+ apply_remote, &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -717,7 +723,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -743,6 +750,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -756,6 +765,25 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'update_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, resolver, slot, subid,
+ apply_remote, tid))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c773a7016b..95157438ce 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2517,8 +2518,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2670,7 +2672,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2695,14 +2698,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2710,10 +2712,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2754,6 +2757,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2766,7 +2771,32 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3264,6 +3294,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3271,7 +3303,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f3909759f8..291d5dde36 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 784f89fe2d..d9333dc962 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
--
2.34.1
v11-0004-Manage-Clock-skew-and-implement-last_update_wins.patchapplication/octet-stream; name=v11-0004-Manage-Clock-skew-and-implement-last_update_wins.patchDownload
From 69d3ac557246355c3999c98ebe349dd6190cbf40 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Fri, 30 Aug 2024 08:55:09 +0530
Subject: [PATCH v11 4/4] Manage Clock skew and implement last_update_wins
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
This patch also implements last_update_wins resolver.
Since conflict resolution for two phase commit transactions
using prepare-timestamp can result in data divergence, this patch
also restricts enabling two_phase and detect_conflict together
for a subscription.
---
src/backend/commands/subscriptioncmds.c | 30 +++-
src/backend/executor/execReplication.c | 68 +++-----
.../replication/logical/applyparallelworker.c | 23 ++-
src/backend/replication/logical/conflict.c | 128 +++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 162 ++++++++++++++++--
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 +++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/conflict.h | 12 +-
src/include/replication/logicalworker.h | 18 ++
src/include/replication/origin.h | 1 +
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
.../subscription/t/034_conflict_resolver.pl | 136 ++++++++++++++-
src/tools/pgindent/typedefs.list | 1 +
16 files changed, 558 insertions(+), 75 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 38dd67487e..5f4e601363 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -448,7 +448,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
*/
static void
parse_subscription_conflict_resolvers(List *stmtresolvers,
- ConflictTypeResolver * resolvers)
+ ConflictTypeResolver * resolvers,
+ bool twophase)
{
ListCell *lc;
@@ -466,6 +467,16 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
/* update the corresponding resolver for the given conflict type */
resolvers[type].resolver = defGetString(defel);
+
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow enabling both together.
+ */
+ if (twophase && strcmp(resolvers[type].resolver, "last_update_wins") == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("Setting any resolver to last_update_wins and %s are mutually exclusive options",
+ "two_phase = true")));
}
}
@@ -635,7 +646,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
check_conflict_detection();
SetDefaultResolvers(conflictResolvers);
- parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers,
+ opts.twophase);
+
+
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1374,6 +1388,15 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable two_phase when a time based resolver is configured")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1631,7 +1654,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
check_conflict_detection();
/* get list of conflict types and resolvers and validate them */
- conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers,
+ sub);
/*
* Update the conflict resolvers for the corresponding
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f944df761c..5b031a462a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -565,10 +565,20 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
- ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote, ItemPointer tupleid)
+ TupleTableSlot *slot, Oid subid, ItemPointer tupleid)
{
+ ConflictResolver resolver;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+
+ /*
+ * Proceed only if the resolver is not set to 'ERROR'; if the resolver is
+ * 'ERROR', the caller will handle it.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type, NULL, NULL,
+ subid);
+ if (resolver == CR_ERROR)
+ return false;
/* Check all the unique indexes for a conflict */
foreach_oid(uniqueidx, conflictindexes)
@@ -584,8 +594,17 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
+ bool apply_remote = false;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type,
+ &apply_remote, NULL, subid);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
@@ -634,8 +653,6 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
@@ -650,23 +667,10 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'insert_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote, &invalidItemPtr))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, slot, subid,
+ &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -750,8 +754,6 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -765,23 +767,9 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'update_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_UPDATE_EXISTS, resolver, slot, subid,
- apply_remote, tid))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, slot, subid, tid))
return;
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..da2857b59a 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,15 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Don't start a new parallel worker if user has either configured max
+ * clock skew or if 'last_update_wins' is configured for any conflict
+ * type. In both cases we need commit timestamp in the beginning.
+ */
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
@@ -696,9 +706,20 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured to last_update_wins, thus it is okay to pass 0 as
+ * origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with
+ * last_update_wins resolver, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 9fd96d4979..e7e7571427 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -49,6 +49,7 @@ static const char *const ConflictTypeNames[] = {
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -74,12 +75,12 @@ static const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -387,6 +388,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -666,6 +672,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -710,6 +725,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -738,7 +789,7 @@ can_create_full_tuple(Relation localrel,
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
List *
-GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+GetAndValidateSubsConflictResolverList(List *stmtresolvers, Subscription *sub)
{
ListCell *lc;
ConflictTypeResolver *CTR = NULL;
@@ -758,6 +809,18 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
*/
validate_conflict_type_and_resolver(CTR->conflict_type, CTR->resolver);
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ */
+ if ((strcmp(CTR->resolver, "last_update_wins") == 0) &&
+ sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver for a subscription that has two_phase enabled",
+ "last_update_wins")));
+
res = lappend(res, CTR);
}
@@ -977,15 +1040,31 @@ RemoveSubscriptionConflictBySubid(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1013,3 +1092,40 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index 419e4814f0..bd8e6f0024 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -155,6 +155,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 95157438ce..18dc3aa332 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,20 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -985,6 +999,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1006,6 +1109,15 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -1063,6 +1175,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1318,7 +1433,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2023,7 +2139,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2076,6 +2193,16 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
+ /*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /* Capture the timestamp (prepare or commit) of the remote transaction */
+ replorigin_session_origin_timestamp = origin_timestamp;
+
/*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
@@ -2181,7 +2308,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
@@ -2507,7 +2635,7 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
{
EPQState epqstate;
- EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+ /* EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL); */
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
@@ -2522,7 +2650,7 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
rel_entry->localindexoid, true,
conflictslot);
- EvalPlanQualEnd(&epqstate);
+ /* EvalPlanQualEnd(&epqstate); */
ExecDropSingleTupleTableSlot(conflictslot);
}
@@ -2738,7 +2866,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
{
TupleTableSlot *newslot;
- resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2811,7 +2940,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* the configured resolver is in favor of applying the change, convert
* UPDATE to INSERT and apply the change.
*/
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -2964,7 +3093,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_DELETE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2993,7 +3123,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* The tuple to be deleted could not be found. Based on resolver
* configured, either skip and log a message or emit an error.
*/
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MySubscription->oid);
@@ -3190,7 +3320,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -3233,7 +3363,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
newslot = table_slot_create(partrel, &estate->es_tupleTable);
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, partrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -4765,6 +4896,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4805,10 +4937,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb4044d6..0ebad6fcab 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 880d76aae0..4a4f54a0c7 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3714,6 +3716,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4991,6 +5020,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..6424432362 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c0b7b91063..3182350e64 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -62,6 +63,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -102,15 +106,19 @@ extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictBySubid(Oid confid);
-extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers,
+ Subscription *sub);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
Oid subid);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
+
#endif
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..7cb03062ac 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261d7e..95b2a5286d 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index d9333dc962..6955e91a1f 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -252,6 +280,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -305,16 +363,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -401,10 +491,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -431,6 +525,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9e951a9e6f..304a22e55a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1565,6 +1565,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
On Fri, Aug 23, 2024 at 10:39 AM shveta malik <shveta.malik@gmail.com> wrote:
On Thu, Aug 22, 2024 at 3:44 PM shveta malik <shveta.malik@gmail.com> wrote:
For clock-skew and timestamp based resolution, if needed, I will post
another email for the design items where suggestions are needed.Please find issues which need some thoughts and approval for
time-based resolution and clock-skew.1)
Time based conflict resolution and two phase transactions:Time based conflict resolution (last_update_wins) is the one
resolution which will not result in data-divergence considering
clock-skew is taken care of. But when it comes to two-phase
transactions, it might not be the case. For two-phase transaction, we
do not have commit timestamp when the changes are being applied. Thus
for time-based comparison, initially it was decided to user prepare
timestamp but it may result in data-divergence. Please see the
example at [1].Example at [1] is a tricky situation, and thus in the initial draft,
we decided to restrict usage of 2pc and CDR together. The plan is:a) During Create subscription, if the user has given last_update_wins
resolver for any conflict_type and 'two_phase' is also enabled, we
ERROR out.
b) During Alter subscription, if the user tries to update resolver to
'last_update_wins' but 'two_phase' is enabled, we error out.Another solution could be to save both prepare_ts and commit_ts. And
when any txn comes for conflict resolution, we first check if
prepare_ts is available, use that else use commit_ts. Availability of
prepare_ts would indicate it was a prepared txn and thus even if it is
committed, we should use prepare_ts for comparison for consistency.
This will have some overhead of storing prepare_ts along with
commit_ts. But if the number of prepared txns are reasonably small,
this overhead should be less.We currently plan to go with restricting 2pc and last_update_wins
together, unless others have different opinions.
Done. v11-004 implements the idea of restricting 2pc and
last_update_wins together.
~~
2)
parallel apply worker and conflict-resolution:
As discussed in [2] (see last paragraph in [2]), for streaming of
in-progress transactions by parallel worker, we do not have
commit-timestamp with each change and thus it makes sense to disable
parallel apply worker with CDR. The plan is to not start parallel
apply worker if 'last_update_wins' is configured for any
conflict_type.
Done.
~~
3)
parallel apply worker and clock skew management:
Regarding clock-skew management as discussed in [3], we will wait for
the local clock to come within tolerable range during 'begin' rather
than before 'commit'. And this wait needs commit-timestamp in the
beginning, thus we plan to restrict starting pa-worker even when
clock-skew related GUCs are configured.
Done. v11 implements it.
Earlier we had restricted both 2pc and parallel worker worker start
when detect_conflict was enabled, but now since detect_conflict
parameter is removed, we will change the implementation to restrict
all 3 above cases when last_update_wins is configured. When the
changes are done, we will post the patch.~~
--
Thanks,
Nisha
On Mon, Aug 26, 2024 at 2:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Thu, Aug 22, 2024 at 3:45 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 21, 2024 at 4:08 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
The patches have been rebased on the latest pgHead following the merge
of the conflict detection patch [1].Thanks for working on patches.
Summarizing the issues which need some suggestions/thoughts.
1)
For subscription based resolvers, currently the syntax implemented is:1a)
CREATE SUBSCRIPTION <subname>
CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);1b)
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2,
conflict_type3 = resolver3,...);Earlier the syntax suggested in [1] was:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname>
CONFLICT RESOLVER 'conflict_resolver1' FOR 'conflict_type1',
CONFLICT RESOLVER 'conflict_resolver2' FOR 'conflict_type2';I think the currently implemented syntax is good as it has less
repetition, unless others think otherwise.~~
2)
For subscription based resolvers, do we need a RESET command to reset
resolvers to default? Any one of below or both?2a) reset all at once:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVERS2b) reset one at a time:
ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER for 'conflict_type';The issue I see here is, to implement 1a and 1b, we have introduced
the 'RESOLVER' keyword. If we want to implement 2a, we will have to
introduce the 'RESOLVERS' keyword as well. But we can come up with
some alternative syntax if we plan to implement these. Thoughts?It makes sense to have a RESET on the lines of (a) and (b). At this
stage, we should do minimal in extending the syntax. How about RESET
CONFLICT RESOLVER ALL for (a)?
Done, v11 implements the suggested RESET command.
~~
--
Thanks,
Nisha
On Wed, Aug 28, 2024 at 10:58 AM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
2)
Currently pg_dump is dumping even the default resolvers configuration.
As an example if I have not changed default configuration for say
sub1, it still dumps all:CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 WITH (....)
CONFLICT RESOLVER (insert_exists = 'error', update_differ =
'apply_remote', update_exists = 'error', update_missing = 'skip',
delete_differ = 'apply_remote', delete_missing = 'skip');I am not sure if we need to dump default resolvers. Would like to know
what others think on this.
Normally, we don't add defaults in the dumped command. For example,
dumpSubscription won't dump the options where the default is
unchanged. We shouldn't do it unless we have a reason for dumping
defaults.
3)
Why in 002_pg_dump.pl we have default resolvers set explicitly?In 003_pg_dump.pl, default resolvers are not set explicitly, that is the regexp to check the pg_dump generated command for creating subscriptions. This is again connected to your 2nd question.
Okay so we may not need this change if we plan to *not *dump defaults
in pg_dump.Another point about 'defaults' is regarding insertion into the
pg_subscription_conflict table. We currently do insert default
resolvers into 'pg_subscription_conflict' even if the user has not
explicitly configured them.
I don't see any problem with it. BTW, if we don't do it, I think
wherever we are referring the resolvers for a conflict, we need some
special handling for default and non-default. Am I missing something?
--
With Regards,
Amit Kapila.
On Fri, Aug 30, 2024 at 12:13 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:58 AM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
2)
Currently pg_dump is dumping even the default resolvers configuration.
As an example if I have not changed default configuration for say
sub1, it still dumps all:CREATE SUBSCRIPTION sub1 CONNECTION '..' PUBLICATION pub1 WITH (....)
CONFLICT RESOLVER (insert_exists = 'error', update_differ =
'apply_remote', update_exists = 'error', update_missing = 'skip',
delete_differ = 'apply_remote', delete_missing = 'skip');I am not sure if we need to dump default resolvers. Would like to know
what others think on this.Normally, we don't add defaults in the dumped command. For example,
dumpSubscription won't dump the options where the default is
unchanged. We shouldn't do it unless we have a reason for dumping
defaults.
Agreed, we should not dump defaults. I had the same opinion.
3)
Why in 002_pg_dump.pl we have default resolvers set explicitly?In 003_pg_dump.pl, default resolvers are not set explicitly, that is the regexp to check the pg_dump generated command for creating subscriptions. This is again connected to your 2nd question.
Okay so we may not need this change if we plan to *not *dump defaults
in pg_dump.Another point about 'defaults' is regarding insertion into the
pg_subscription_conflict table. We currently do insert default
resolvers into 'pg_subscription_conflict' even if the user has not
explicitly configured them.I don't see any problem with it.
Yes, no problem
BTW, if we don't do it, I think
wherever we are referring the resolvers for a conflict, we need some
special handling for default and non-default.
Yes, we will need special handling in such a case. Thus we shall go
with inserting defaults.
Am I missing something?
No, I just wanted to know others' opinions, so I asked.
thanks
Shveta
On Wed, Aug 28, 2024 at 4:07 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
The review is WIP. Please find a few comments on patch001.
1)
logical-repliction.sgmlL+ Additional logging is triggered for specific conflict_resolvers.
Users can also configure conflict_types while creating the
subscription. Refer to section CONFLICT RESOLVERS for details on
conflict_types and conflict_resolvers.Can we please change it to:
Additional logging is triggered in various conflict scenarios, each
identified as a conflict type. Users have the option to configure a
conflict resolver for each conflict type when creating a subscription.
For more information on the conflict types detected and the supported
conflict resolvers, refer to the section <CONFLICT RESOLVERS>2)
SetSubConflictResolver+ for (type = 0; type < resolvers_cnt; type++)
'type' does not look like the correct name here. The variable does not
state conflict_type, it is instead a resolver-array-index, so please
rename accordingly. Maybe idx or res_idx?3)
CreateSubscription():+ if (stmt->resolvers) + check_conflict_detection();3a) We can have a comment saying warn users if prerequisites are not met.
3b) Also, I do not find the name 'check_conflict_detection'
appropriate. One suggestion could be
'conf_detection_check_prerequisites' (similar to
replorigin_check_prerequisites)3c) We can move the below comment after check_conflict_detection() as
it makes more sense there.
/*
* Parse and check conflict resolvers. Initialize with default values
*/4)
Should we allow repetition/duplicates of 'conflict_type=..' in CREATE
and ALTER SUB? As an example:
ALTER SUBSCRIPTION sub1 CONFLICT RESOLVER (insert_exists =
'apply_remote', insert_exists = 'error');Such a repetition works for Create-Sub but gives some internal error
for alter-sub. (ERROR: tuple already updated by self). Behaviour
should be the same for both. And if we give an error, it should be
some user understandable one. But I would like to know the opinions of
others. Shall it give an error or the last one should be accepted as
valid configuration in case of repetition?
I have tried the below statement to check existing behavior:
create subscription sub1 connection 'dbname=postgres' publication pub1
with (streaming = on, streaming=off);
ERROR: conflicting or redundant options
LINE 1: ...=postgres' publication pub1 with (streaming = on, streaming=...
So duplicate options are not allowed. If we see any challenges to
follow same for resolvers then we can discuss but it seems better to
follow the existing behavior of other subscription options.
Also, the behavior for CREATE/ALTER should be the same.
--
With Regards,
Amit Kapila.
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664
+ | ALTER SUBSCRIPTION name RESET CONFLICT
RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind =
ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst
{ $$ = $1; }
+ | NULL_P
{ $$ = NULL; }
;
May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;
2) Conflict resolver is not shown in describe command:
postgres=# \dRs+
List of subscriptions
Name | Owner | Enabled | Publication | Binary | Streaming |
Two-phase commit | Disable on error | Origin | Password required | Run
as owner? | Failover | Synchronous commit | Conninfo
| Skip LSN
------+---------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+----------------------------------
--------+----------
sub1 | vignesh | t | {pub1} | f | off | d
| f | any | t | f
| f | off | dbname=postgres host=localhost po
rt=5432 | 0/0
sub2 | vignesh | t | {pub1} | f | off | d
| f | any | t | f
| f | off | dbname=postgres host=localhost po
rt=5432 | 0/0
(2 rows)
3) Tab completion is not handled to include Conflict resolver:
postgres=# alter subscription sub1
ADD PUBLICATION CONNECTION DISABLE DROP
PUBLICATION ENABLE OWNER TO REFRESH
PUBLICATION RENAME TO SET SKIP (
Regards,
Vignesh
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) Updated conflict type names in accordance with the recent commit[1] as -
update_differ --> update_origin_differs
delete_differ --> delete_origin_differs2) patch-001:
- Implemented the RESET command to restore the default resolvers as
suggested in pt.2a & 2b in [2]
Few comments on 0001 patch:
1) Currently create subscription has WITH option before conflict
resolver, I felt WITH option can be after CONNECTION, PUBLICATION and
CONFLICT RESOLVER option and WITH option at the end:
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst
PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION
name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10696,6 +10702,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
2) Case sensitive:
2.a) Should conflict type be case insensitive:
CREATE SUBSCRIPTION sub3 CONNECTION 'dbname=postgres host=localhost
port=5432' PUBLICATION pub1 with (copy_data= true) CONFLICT RESOLVER
("INSERT_EXISTS" = 'error');
ERROR: INSERT_EXISTS is not a valid conflict type
In few other places it is not case sensitive:
create publication pub1 with ( PUBLISH= 'INSERT,UPDATE,delete');
set log_min_MESSAGES TO warning ;
2.b) Similarly in case of conflict resolver too:
CREATE SUBSCRIPTION sub3 CONNECTION 'dbname=postgres host=localhost
port=5432' PUBLICATION pub1 with (copy_data= true) CONFLICT RESOLVER
("insert_exists" = 'erroR');
ERROR: erroR is not a valid conflict resolver
3) Since there is only one key used to search, we can remove nkeys
variable and directly specify as 1:
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
4) Currently we are including CONFLICT RESOLVER even if a subscription
with default CONFLICT RESOLVER is created, we can add the CONFLICT
RESOLVER option only for non-default subscription option:
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ PQExpBuffer InQry = createPQExpBuffer();
+ PGresult *res;
+ int i_confrtype;
+ int i_confrres;
+
+ /* get the conflict types and their resolvers from the
catalog */
+ appendPQExpBuffer(InQry,
+ "SELECT confrtype, confrres "
+ "FROM
pg_catalog.pg_subscription_conflict"
+ " WHERE confsubid =
%u;\n", subinfo->dobj.catId.oid);
+ res = ExecuteSqlQuery(fout, InQry->data, PGRES_TUPLES_OK);
+
+ i_confrtype = PQfnumber(res, "confrtype");
+ i_confrres = PQfnumber(res, "confrres");
+
+ if (PQntuples(res) > 0)
+ {
+ appendPQExpBufferStr(query, ") CONFLICT RESOLVER (");
5) Should remote_apply be apply_remote here as this is what is
specified in code:
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
6) I think this should be "It is the default resolver for update_origin_differs"
6.a)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
6.b)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-update-differ">
+ <term><literal>update_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
6.c)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
6.d)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
Similarly this change should be done in other places too.
7)
7.a) Should delete_differ be changed to delete_origin_differs as that
is what is specified in the subscription commands:
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and
resolution could be incomplete due to disabled
track_commit_timestamp"),
+ errdetail("Conflicts update_differ and
delete_differ cannot be detected, "
+ "and the origin and
commit timestamp for the local row will not be logged."));
7.b) similarly here too:
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-delete-differ">
+ <term><literal>delete_differ</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was
previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link
linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is
always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
Similarly this change should be done in other places too.
8) ConflictTypeResolver should be added to typedefs.list to resolve
the pgindent issues:
8.a)
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+
ConflictTypeResolver * resolvers)
8.b) Similarly FormData_pg_subscription_conflict should also be added:
} FormData_pg_subscription_conflict;
/* ----------------
* Form_pg_subscription_conflict corresponds to a pointer to a row with
* the format of pg_subscription_conflict relation.
* ----------------
*/
typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
Regards,
Vignesh
On Thu, Aug 29, 2024 at 2:50 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 4:07 PM shveta malik <shveta.malik@gmail.com>
wrote:On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com>
wrote:
The review is WIP. Please find a few comments on patch001.
More comments on ptach001 in continuation of previous comments:
Thank you for your feedback, Shveta. I've addressed both sets of comments
you provided. Additionally, I've revised the logic for how `pg_dump`
retrieves conflict resolver information and implemented a smarter approach
to avoid including default resolvers in the `CREATE SUBSCRIPTION` command.
To achieve this, I had to duplicate some of the conflict resolver
definitions in the `pg_dump.h` header, as including `conflict.h` directly
introduced too many dependencies with other headers.
Thanks to Nisha for separating 004 patch into two - 004(last_update_wins)
and 005(clock-skew).
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v12-0005-Implements-Clock-skew-management-between-nodes.patchapplication/octet-stream; name=v12-0005-Implements-Clock-skew-management-between-nodes.patchDownload
From f4bc1c1462003e78716cf86c39ea62e0e1ac89fb Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 6 Sep 2024 03:26:24 -0400
Subject: [PATCH v12 5/5] Implements Clock-skew management between nodes.
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
---
.../replication/logical/applyparallelworker.c | 22 +++-
src/backend/replication/logical/worker.c | 125 ++++++++++++++++++++-
src/backend/utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 +++++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/logicalworker.h | 18 +++
src/include/replication/worker_internal.h | 2 +-
src/include/utils/timestamp.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
9 files changed, 209 insertions(+), 10 deletions(-)
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 6cbfab0..809dab3 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -314,15 +314,16 @@ pa_can_start(void)
return false;
/*
- * Do not start a new parallel worker if 'last_update_wins' is configured
- * for any conflict type, as we need the commit timestamp in the
- * beginning.
+ * Do not start a new parallel worker if either max clock skew or
+ * 'last_update_wins' is configured for any conflict type. In both of the
+ * cases we need the commit timestamp in the beginning.
*
* XXX: To lift this restriction, we could write the changes to a file
* when a conflict is detected, and then at the commit time, let the
* remaining changes be applied by the apply worker.
*/
- if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ CheckIfSubHasTimeStampResolver(MySubscription->oid))
return false;
return true;
@@ -709,9 +710,20 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured to last_update_wins, thus it is okay to pass 0 as
+ * origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with
+ * last_update_wins resolver, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 00bf47c..da6702f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,6 +319,20 @@ static uint32 parallel_stream_nchanges = 0;
bool InitializingApplyWorker = false;
/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
+/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
* Once we start skipping changes, we don't stop it until we skip all changes of
@@ -986,6 +1000,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
}
/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
+/*
* Handle BEGIN message.
*/
static void
@@ -1007,6 +1110,9 @@ apply_handle_begin(StringInfo s)
pgstat_report_activity(STATE_RUNNING, NULL);
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
/*
* Capture the commit timestamp of the remote transaction for time based
* conflict resolution purpose.
@@ -1069,6 +1175,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1324,7 +1433,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2029,7 +2139,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2083,6 +2194,13 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
/*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
*/
@@ -2187,7 +2305,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb404..0ebad6f 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309d..c768a119 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3714,6 +3716,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4992,6 +5021,17 @@ struct config_enum ConfigureNamesEnum[] =
},
{
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
+ {
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc..6424432 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d..7cb0306 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261..95b2a52 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03e..53b828d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index df3f336..7d20a3c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1566,6 +1566,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
1.8.3.1
v12-0004-Implements-last_update_wins-conflict-resolver.patchapplication/octet-stream; name=v12-0004-Implements-last_update_wins-conflict-resolver.patchDownload
From 8d2aeb2d07c49b9df127f8d1668db6d14c7e2586 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 6 Sep 2024 03:15:33 -0400
Subject: [PATCH v12 4/5] Implements last_update_wins conflict resolver.
This resolver is applicable for conflict types: insert_exists, update_exists,
update_origin_differs and delete_origin_differs.
For these conflicts, when the resolver is set to last_update_wins,
the timestamps of the remote and local conflicting tuple are compared to
determine whether to apply or ignore the remote changes.
The GUC track_commit_timestamp must be enabled to support this resolver.
Since conflict resolution for two phase commit transactions using
prepare-timestamp can result in data divergence, this patch restricts
enabling both two_phase and the last_update_wins resolver together
for a subscription.
The patch also restrict starting a parallel apply worker if resolver is set
to last_update_wins for any conflict type.
---
src/backend/commands/subscriptioncmds.c | 37 +++++-
src/backend/executor/execReplication.c | 68 +++++------
.../replication/logical/applyparallelworker.c | 13 ++
src/backend/replication/logical/conflict.c | 131 +++++++++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 30 +++--
src/include/replication/conflict.h | 11 +-
src/include/replication/origin.h | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 136 +++++++++++++++++++--
9 files changed, 361 insertions(+), 67 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 66cbbc0..85f045b 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -446,7 +446,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
*/
static void
parse_subscription_conflict_resolvers(List *stmtresolvers,
- ConflictTypeResolver * resolvers)
+ ConflictTypeResolver * resolvers,
+ bool twophase)
{
ListCell *lc;
@@ -464,6 +465,21 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
/* update the corresponding resolver for the given conflict type */
resolvers[type].resolver = defGetString(defel);
+
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow enabling both together.
+ *
+ * XXX: An alternative solution idea is that if a conflict is detected
+ * and the resolution strategy is last_update_wins, then start writing
+ * all the changes to a file similar to what we do for streaming mode.
+ * Once commit_prepared arrives, we will read and apply the changes.
+ */
+ if (twophase && strcmp(resolvers[type].resolver, "last_update_wins") == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("Setting any resolver to last_update_wins and %s are mutually exclusive options",
+ "two_phase = true")));
}
}
@@ -634,7 +650,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
conf_detection_check_prerequisites();
/* Parse and check conflict resolvers. */
- parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers,
+ opts.twophase);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1373,6 +1390,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is
+ * implemented.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable two_phase when a time based resolver is configured")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1630,7 +1660,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
conf_detection_check_prerequisites();
/* get list of conflict types and resolvers and validate them */
- conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers,
+ sub);
/*
* Update the conflict resolvers for the corresponding
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f944df7..5b031a4 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -565,10 +565,20 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
- ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote, ItemPointer tupleid)
+ TupleTableSlot *slot, Oid subid, ItemPointer tupleid)
{
+ ConflictResolver resolver;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+
+ /*
+ * Proceed only if the resolver is not set to 'ERROR'; if the resolver is
+ * 'ERROR', the caller will handle it.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type, NULL, NULL,
+ subid);
+ if (resolver == CR_ERROR)
+ return false;
/* Check all the unique indexes for a conflict */
foreach_oid(uniqueidx, conflictindexes)
@@ -584,8 +594,17 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
+ bool apply_remote = false;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type,
+ &apply_remote, NULL, subid);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
@@ -634,8 +653,6 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
@@ -650,23 +667,10 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'insert_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote, &invalidItemPtr))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, slot, subid,
+ &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -750,8 +754,6 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -765,23 +767,9 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'update_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_UPDATE_EXISTS, resolver, slot, subid,
- apply_remote, tid))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, slot, subid, tid))
return;
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c..6cbfab0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,18 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Do not start a new parallel worker if 'last_update_wins' is configured
+ * for any conflict type, as we need the commit timestamp in the
+ * beginning.
+ *
+ * XXX: To lift this restriction, we could write the changes to a file
+ * when a conflict is detected, and then at the commit time, let the
+ * remaining changes be applied by the apply worker.
+ */
+ if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 6f298ae..8ed56b7 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -50,6 +50,7 @@ static const char *const ConflictTypeNames[] = {
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -75,12 +76,12 @@ static const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -390,6 +391,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -671,6 +677,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -716,6 +731,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
}
/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
+/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
*/
@@ -743,7 +794,7 @@ can_create_full_tuple(Relation localrel,
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
List *
-GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+GetAndValidateSubsConflictResolverList(List *stmtresolvers, Subscription *sub)
{
ListCell *lc;
ConflictTypeResolver *conftyperesolver = NULL;
@@ -774,6 +825,21 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
conftyperesolver->conflict_type,
conftyperesolver->resolver);
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is implemented.
+ */
+ if ((strcmp(conftyperesolver->resolver, "last_update_wins") == 0) &&
+ sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver for a subscription that has two_phase enabled",
+ "last_update_wins")));
+
/* Add the conflict type to the list of seen types */
conflictTypes = lappend(conflictTypes, (void *) conftyperesolver->conflict_type);
@@ -1010,15 +1076,31 @@ RemoveSubscriptionConflictResolvers(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1046,3 +1128,40 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index baf696d..3094030 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -159,6 +159,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9515743..00bf47c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1006,6 +1006,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -2738,7 +2744,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
{
TupleTableSlot *newslot;
- resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2811,7 +2818,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* the configured resolver is in favor of applying the change, convert
* UPDATE to INSERT and apply the change.
*/
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -2964,7 +2971,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_DELETE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2993,7 +3001,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* The tuple to be deleted could not be found. Based on resolver
* configured, either skip and log a message or emit an error.
*/
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MySubscription->oid);
@@ -3190,7 +3198,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -3233,7 +3241,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
newslot = table_slot_create(partrel, &estate->es_tupleTable);
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, partrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -4765,6 +4774,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4805,10 +4815,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 2c9b61c..c36b942 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -70,6 +71,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -110,17 +114,20 @@ extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictResolvers(Oid confid);
-extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers,
+ Subscription *sub);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
Oid subid);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
#endif
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9..dcbbbdf 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index d9333dc..6955e91 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -252,6 +280,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -305,16 +363,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -401,10 +491,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -431,6 +525,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
--
1.8.3.1
v12-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v12-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From e574f8dcc25ad3ebba5f8024d0ba323f78c7b906 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 6 Sep 2024 03:08:36 -0400
Subject: [PATCH v12 3/5] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 44 ++++++++--
src/backend/replication/logical/worker.c | 84 ++++++++++++++++---
src/include/executor/executor.h | 3 +-
src/test/subscription/t/034_conflict_resolver.pl | 100 +++++++++++++++++++++++
4 files changed, 209 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4059707..f944df7 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -566,7 +566,7 @@ static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote)
+ Oid subid, bool apply_remote, ItemPointer tupleid)
{
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
@@ -579,7 +579,7 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
* otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -636,6 +636,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool conflict = false;
ConflictResolver resolver;
bool apply_remote = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -656,11 +657,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
NULL, subid);
- /* Check for conflict and return to caller for resolution if found */
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'insert_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
if (resolver != CR_ERROR &&
has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote))
+ apply_remote, &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -717,7 +723,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -743,6 +750,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -756,6 +765,25 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'update_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, resolver, slot, subid,
+ apply_remote, tid))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c773a70..9515743 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2517,8 +2518,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2670,7 +2672,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2695,14 +2698,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2710,10 +2712,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2754,6 +2757,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2766,7 +2771,32 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3264,6 +3294,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3271,7 +3303,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f390975..291d5dd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 784f89f..d9333dc 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
--
1.8.3.1
v12-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v12-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 46443d3325441abfcd6cfdaf310d1415b62e769a Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 6 Sep 2024 03:07:34 -0400
Subject: [PATCH v12 2/5] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 68 ++-
src/backend/replication/logical/conflict.c | 223 +++++++--
src/backend/replication/logical/worker.c | 370 +++++++++++----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 13 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 581 +++++++++++++++++++++++
7 files changed, 1115 insertions(+), 146 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc..4059707 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,14 +550,59 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
+ TupleTableSlot **conflictslot, ConflictType type,
+ ConflictResolver resolver, TupleTableSlot *slot,
+ Oid subid, bool apply_remote)
+{
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
*
@@ -565,7 +610,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -588,6 +634,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -601,6 +649,20 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index b925479..6f298ae 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -30,9 +30,10 @@
#include "executor/executor.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/worker_internal.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
-#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -99,12 +100,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -147,8 +150,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -167,13 +170,22 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
+
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
@@ -182,13 +194,14 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -256,17 +269,24 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -277,13 +297,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -293,47 +314,62 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"),
+ applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -640,6 +676,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
+/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
@@ -903,3 +1002,47 @@ RemoveSubscriptionConflictResolvers(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 0fb577d..c773a70 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2704,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2727,47 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2777,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2911,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2928,51 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3020,19 +3105,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3058,6 +3145,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3070,47 +3160,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3122,23 +3250,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3179,12 +3336,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3200,22 +3365,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebf..f390975 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 296270a..2c9b61c 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -96,12 +98,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -113,5 +117,10 @@ extern ConflictType validate_conflict_type_and_resolver(const char *conflict_typ
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7..00ade29 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000..784f89f
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,581 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
1.8.3.1
v12-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v12-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From abf142c1ccc6df520da6413c0ca8677d52948059 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 6 Sep 2024 02:58:24 -0400
Subject: [PATCH v12 1/5] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for reseting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 86 +----
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 12 +
doc/src/sgml/ref/create_subscription.sgml | 177 +++++++++++
src/backend/commands/subscriptioncmds.c | 87 +++++-
src/backend/parser/gram.y | 48 ++-
src/backend/replication/logical/conflict.c | 414 +++++++++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 96 +++++-
src/bin/pg_dump/pg_dump.h | 118 ++++++-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_subscription_conflict.h | 55 ++++
src/include/nodes/parsenodes.h | 6 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 51 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 107 +++++++
src/test/regress/sql/subscription.sql | 47 +++
18 files changed, 1218 insertions(+), 105 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index df62eb4..d6dcad3 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,83 +1582,15 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
+ Additional logging is triggered in various conflict scenarios, each identified as a
+ conflict type, and the conflict statistics are collected (displayed in the
<link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
+ Users have the option to configure a <literal>conflict_resolver</literal> for each
+ <literal>conflict_type</literal> when creating a subscription.
+ For more information on the <literal>conflict_types</literal> detected and the supported
+ <literal>conflict_resolvers</literal>, refer to section
+ <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>.
+
Note that there are other conflict scenarios, such as exclusion constraint
violations. Currently, we do not provide additional details for them in the
log.
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 933de6f..6ead5c3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d..e7b39f2
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
</synopsis>
</refsynopsisdiv>
@@ -345,6 +346,17 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d94..3327a19
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -25,6 +25,7 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,182 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_differ</literal> and <literal>delete_differ</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc63..66cbbc0 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -28,6 +29,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -37,6 +39,7 @@
#include "miscadmin.h"
#include "nodes/makefuncs.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -113,7 +116,6 @@ static void ReportSlotConnectionError(List *rstates, Oid subid, char *slotname,
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
-
/*
* Common option parsing function for CREATE and ALTER SUBSCRIPTION commands.
*
@@ -440,6 +442,32 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver * resolvers)
+{
+ ListCell *lc;
+
+ /* first initialise the resolvers with default values */
+ SetDefaultResolvers(resolvers);
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+
+ /* validate the conflict type and resolver */
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* update the corresponding resolver for the given conflict type */
+ resolvers[type].resolver = defGetString(defel);
+ }
+}
+
+/*
* Add publication names from the list to a string.
*/
static void
@@ -583,6 +611,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
/*
* Parse and check options.
@@ -598,6 +627,16 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/*
+ * Warn users if prerequisites are not met.
+ * Initialize with default values.
+ */
+ if (stmt->resolvers)
+ conf_detection_check_prerequisites();
+
+ /* Parse and check conflict resolvers. */
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
+ /*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
* CREATE SUBSCRIPTION inside a transaction block if creating a
@@ -723,6 +762,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1623,46 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ conf_detection_check_prerequisites();
+
+ /* get list of conflict types and resolvers and validate them */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
+ {
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+
+ /* remove existing conflict resolvers */
+ RemoveSubscriptionConflictResolvers(subid);
+
+ /*
+ * create list of conflict resolvers and set them in the
+ * catalog
+ */
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER:
+ {
+ /*
+ * reset the conflict resolver for this conflict type to its
+ * default
+ */
+ ResetConflictResolver(subid, stmt->conflict_type);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1914,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 84cef57..be87487 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -612,6 +612,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -769,7 +770,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8754,6 +8755,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10688,7 +10694,7 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10696,6 +10702,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
}
;
@@ -10802,6 +10809,39 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
+ | NULL_P { $$ = NULL; }
;
/*****************************************************************************
@@ -17725,6 +17765,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18353,6 +18394,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff62..b925479 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,13 +15,27 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -32,6 +46,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -489,3 +552,354 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (strcmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (strcmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *conftyperesolver = NULL;
+ List *res = NIL;
+ List *conflictTypes = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+
+ /* Check if the conflict type already exists in the list */
+ if (list_member(conflictTypes, defel->defname))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+ }
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type = defel->defname;
+ conftyperesolver->resolver = defGetString(defel);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+
+ /* Add the conflict type to the list of seen types */
+ conflictTypes = lappend(conflictTypes, (void *) conftyperesolver->conflict_type);
+
+ /* Add the validated ConflictTypeResolver to the result list */
+ res = lappend(res, conftyperesolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *CTR = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(CTR->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] = CStringGetTextDatum(CTR->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ CTR->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver conflictResolver;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ Relation pg_subscription_conflict;
+ HeapTuple oldtup,
+ newtup;
+ bool valid = false;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Get the index for this conflict_type */
+ for (idx = CT_MIN; idx <= CT_MAX; idx++)
+ {
+ if (strcmp(ConflictTypeNames[idx], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* get the default resolver for this conflict_type */
+ conflictResolver.resolver = ConflictResolverNames[ConflictTypeDefaultResolvers[idx]];
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u", conflict_type, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confrres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /* check if current resolver is the default one, if not update it */
+ if (strcmp(cur_conflict_res, conflictResolver.resolver) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conflictResolver.resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_differ and delete_differ cannot be detected, "
+ "and the origin and commit timestamp for the local row will not be logged."));
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int idx;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+
+ for (idx = 0; idx < resolvers_cnt; idx++)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[idx].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[idx].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 546e7e4..27f9f30 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4827,7 +4827,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
- PGresult *res;
+ PQExpBuffer confQuery;
+ PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4848,7 +4850,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5009,6 +5013,67 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+ for (j = 0; j < ntuples; j++)
+ {
+ char *confType = PQgetvalue(confRes, j, 0);
+ char *confResVal = PQgetvalue(confRes, j, 1);
+
+ if (strcmp(confType, "delete_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "delete_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "insert_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5254,6 +5319,33 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ ConflictType idx;
+ bool first_resolver = true;
+
+ for (idx = 0; idx < CONFLICT_NUM_TYPES; idx++)
+ {
+ if (strcmp(subinfo->conflict_resolver[idx].resolver,
+ ConflictResolverNames[ConflictTypeDefaultResolvers[idx]]) != 0)
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query,", %s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ }
+ }
+
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 0b7d21b..13625c6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -654,24 +654,112 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+#define CONFLICT_NUM_TYPES 6
+
+/*
+ * Conflict types that could occur while applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_ORIGIN_DIFFERS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_ORIGIN_DIFFERS,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /*
+ * Other conflicts, such as exclusion constraint violations, involve more
+ * complex rules than simple equality checks. These conflicts are left for
+ * future improvements.
+ */
+} ConflictType;
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+}ConflictResolver;
+
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+}ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
- const char *rolname;
- char *subenabled;
- char *subbinary;
- char *substream;
- char *subtwophasestate;
- char *subdisableonerr;
- char *subpasswordrequired;
- char *subrunasowner;
- char *subconninfo;
- char *subslotname;
- char *subsynccommit;
- char *subpublications;
- char *suborigin;
- char *suboriginremotelsn;
- char *subfailover;
+ const char *rolname;
+ char *subenabled;
+ char *subbinary;
+ char *substream;
+ char *subtwophasestate;
+ char *subdisableonerr;
+ char *subpasswordrequired;
+ char *subrunasowner;
+ char *subconninfo;
+ char *subslotname;
+ char *subsynccommit;
+ char *subpublications;
+ char *suborigin;
+ char *suboriginremotelsn;
+ char *subfailover;
+ ConflictTypeResolver conflict_resolver[CONFLICT_NUM_TYPES];
} SubscriptionInfo;
/*
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a..f2611c1 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1da..959e1d9 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000..211fcf7
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict * Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_sub_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_sub_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 124d853..bcb9a90 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4188,6 +4188,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4200,6 +4201,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4210,6 +4214,8 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f865907..d661a06 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -375,6 +375,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677..296270a 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -50,6 +50,47 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +103,15 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictResolvers(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void ResetConflictResolver(Oid subid, char *conflict_type);
+extern void conf_detection_check_prerequisites(void);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb89..42bf2cc 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..5ee69ab 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -401,6 +401,113 @@ HINT: To initiate replication, you must manually create the replication slot, e
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH
+ (connect = false) CONFLICT RESOLVER (update_missing = 'apply_remote');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | keep_local
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_differ and delete_differ cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
ERROR: disable_on_error requires a Boolean value
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..689c8c5 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,53 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = foo);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH
+ (connect = false) CONFLICT RESOLVER (update_missing = 'apply_remote');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false) CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
--
1.8.3.1
On Thu, Aug 29, 2024 at 4:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
On Fri, Aug 23, 2024 at 10:39 AM shveta malik <shveta.malik@gmail.com> wrote:
Please find issues which need some thoughts and approval for
time-based resolution and clock-skew.1)
Time based conflict resolution and two phase transactions:Time based conflict resolution (last_update_wins) is the one
resolution which will not result in data-divergence considering
clock-skew is taken care of. But when it comes to two-phase
transactions, it might not be the case. For two-phase transaction, we
do not have commit timestamp when the changes are being applied. Thus
for time-based comparison, initially it was decided to user prepare
timestamp but it may result in data-divergence. Please see the
example at [1].Example at [1] is a tricky situation, and thus in the initial draft,
we decided to restrict usage of 2pc and CDR together. The plan is:a) During Create subscription, if the user has given last_update_wins
resolver for any conflict_type and 'two_phase' is also enabled, we
ERROR out.
b) During Alter subscription, if the user tries to update resolver to
'last_update_wins' but 'two_phase' is enabled, we error out.Another solution could be to save both prepare_ts and commit_ts. And
when any txn comes for conflict resolution, we first check if
prepare_ts is available, use that else use commit_ts. Availability of
prepare_ts would indicate it was a prepared txn and thus even if it is
committed, we should use prepare_ts for comparison for consistency.
This will have some overhead of storing prepare_ts along with
commit_ts. But if the number of prepared txns are reasonably small,
this overhead should be less.Yet another idea is that if the conflict is detected and the
resolution strategy is last_update_wins then from that point we start
writing all the changes to the file similar to what we do for
streaming mode and only once commit_prepared arrives, we will read and
apply changes. That will solve this problem.We currently plan to go with restricting 2pc and last_update_wins
together, unless others have different opinions.Sounds reasonable but we should add comments on the possible solution
like the one I have mentioned so that we can extend it afterwards.
Done, v12-004 patch has the comments for the possible solution.
~~
2)
parallel apply worker and conflict-resolution:
As discussed in [2] (see last paragraph in [2]), for streaming of
in-progress transactions by parallel worker, we do not have
commit-timestamp with each change and thus it makes sense to disable
parallel apply worker with CDR. The plan is to not start parallel
apply worker if 'last_update_wins' is configured for any
conflict_type.The other idea is that we can let the changes written to file if any
conflict is detected and then at commit time let the remaining changes
be applied by apply worker. This can introduce some complexity, so
similar to two_pc we can extend this functionality later.
v12-004 patch has the comments to extend it later.
~~
3)
parallel apply worker and clock skew management:
Regarding clock-skew management as discussed in [3], we will wait for
the local clock to come within tolerable range during 'begin' rather
than before 'commit'. And this wait needs commit-timestamp in the
beginning, thus we plan to restrict starting pa-worker even when
clock-skew related GUCs are configured.Earlier we had restricted both 2pc and parallel worker worker start
when detect_conflict was enabled, but now since detect_conflict
parameter is removed, we will change the implementation to restrict
all 3 above cases when last_update_wins is configured. When the
changes are done, we will post the patch.At this stage, we are not sure how we want to deal with clock skew.
There is an argument that clock-skew should be handled outside the
database, so we can probably have the clock-skew-related stuff in a
separate patch.
Separated the clock-skew related code in v12-005 patch.
--
Thanks,
Nisha
On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian <itsajin@gmail.com> wrote:
Thank you for your feedback, Shveta. I've addressed both sets of comments you provided.
Thanks for the patches. I am reviewing v12-patch001, it is WIP. But
please find first set of comments:
1)
src/sgml/logical-replication.sgml:
+ Users have the option to configure a conflict_resolver
Full stop for previous line is missing.
2)
+ For more information on the conflict_types detected and the
supported conflict_resolvers, refer to section CONFLICT RESOLVERS.
We may change to :
For more information on the supported conflict_types and
conflict_resolvers, refer to section CONFLICT RESOLVERS.
3)
src/backend/commands/subscriptioncmds.c:
Line removed. This change is not needed.
static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
-
4)
Let's stick to the same comments format as the rest of the file i.e.
first letter in caps.
+ /* first initialise the resolvers with default values */
first --> First
initialise --> initialize
Same for below comments:
+ /* validate the conflict type and resolver */
+ /* update the corresponding resolver for the given conflict type */
Please verify the rest of the file for the same.
5)
Please add below in header of parse_subscription_conflict_resolvers
(similar to parse_subscription_options):
* This function will report an error if mutually exclusive options
are specified.
6)
+ * Warn users if prerequisites are not met.
+ * Initialize with default values.
+ */
+ if (stmt->resolvers)
+ conf_detection_check_prerequisites();
+
Would it be better to move the above call inside
parse_subscription_conflict_resolvers(), then we will have all
resolver related stuff at one place?
Irrespective of whether we move it or not, please remove 'Initialize
with default values.' from above as that is now not done here.
thanks
Shveta
On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian <itsajin@gmail.com> wrote:
On Thu, Aug 29, 2024 at 2:50 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 4:07 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Aug 28, 2024 at 10:30 AM Ajin Cherian <itsajin@gmail.com> wrote:
The review is WIP. Please find a few comments on patch001.
More comments on ptach001 in continuation of previous comments:
Thank you for your feedback, Shveta. I've addressed both sets of comments you provided.
Thanks for the patches. I tested the v12-0001 patch, and here are my comments:
1) An unexpected error occurs when attempting to alter the resolver
for multiple conflict_type(s) in ALTER SUB...CONFLICT RESOLVER
command. See below examples :
postgres=# alter subscription sub2 CONFLICT RESOLVER
(update_exists=keep_local, delete_missing=error,
update_origin_differs=error);
ERROR: unrecognized node type: 1633972341
postgres=# alter subscription sub2 CONFLICT RESOLVER (
update_origin_differs=error, update_exists=error);
ERROR: unrecognized node type: 1633972341
postgres=# alter subscription sub2 CONFLICT RESOLVER (
delete_origin_differs=error, delete_missing=error);
ERROR: unrecognized node type: 1701602660
postgres=# alter subscription sub2 CONFLICT RESOLVER
(update_exists=keep_local, delete_missing=error);
ALTER SUBSCRIPTION
-- It appears that the error occurs only when at least two conflict
types belong to the same category, either UPDATE or DELETE.
2) Given the above issue, it would be beneficial to add a test in
subscription.sql to cover cases where all valid conflict types are set
with appropriate resolvers in both the ALTER and CREATE commands.
Thanks,
Nisha
On Mon, Sep 9, 2024 at 2:58 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian <itsajin@gmail.com> wrote:
Thank you for your feedback, Shveta. I've addressed both sets of comments you provided.
Thanks for the patches. I am reviewing v12-patch001, it is WIP. But
please find first set of comments:
It will be good if we can use parse_subscription_conflict_resolvers()
from both CREATE and ALTER flow instead of writing different functions
for both the flows. Please review once to see this feasibility.
thanks
Shveta
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com>
wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664
+ | ALTER SUBSCRIPTION name RESET CONFLICT
RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+
makeNode(AlterSubscriptionStmt);
+
+ n->kind =
ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst
{ $$ = $1; }
+ | NULL_P
{ $$ = NULL; }
;
May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;
Fixed.
2) Conflict resolver is not shown in describe command:
postgres=# \dRs+
List of subscriptions
Name | Owner | Enabled | Publication | Binary | Streaming |
Two-phase commit | Disable on error | Origin | Password required | Run
as owner? | Failover | Synchronous commit | Conninfo
| Skip LSN
------+---------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+----------------------------------
--------+----------
sub1 | vignesh | t | {pub1} | f | off | d
| f | any | t | f
| f | off | dbname=postgres host=localhost po
rt=5432 | 0/0
sub2 | vignesh | t | {pub1} | f | off | d
| f | any | t | f
| f | off | dbname=postgres host=localhost po
rt=5432 | 0/0
(2 rows)
Fixed.
3) Tab completion is not handled to include Conflict resolver:
postgres=# alter subscription sub1
ADD PUBLICATION CONNECTION DISABLE DROP
PUBLICATION ENABLE OWNER TO REFRESH
PUBLICATION RENAME TO SET SKIP (
Fixed
On Tue, Sep 3, 2024 at 8:33 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com>
wrote:
Here is the v11 patch-set. Changes are:
1) Updated conflict type names in accordance with the recent
commit[1] as -
update_differ --> update_origin_differs
delete_differ --> delete_origin_differs2) patch-001:
- Implemented the RESET command to restore the default resolvers as
suggested in pt.2a & 2b in [2]
Few comments on 0001 patch:
1) Currently create subscription has WITH option before conflict
resolver, I felt WITH option can be after CONNECTION, PUBLICATION and
CONFLICT RESOLVER option and WITH option at the end:
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst
PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION
name_list opt_definition opt_resolver_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
@@ -10696,6 +10702,7 @@ CreateSubscriptionStmt:
n->conninfo = $5;
n->publication = $7;
n->options = $8;
+ n->resolvers = $9;
$$ = (Node *) n;
Changed as suggested.
2) Case sensitive:
2.a) Should conflict type be case insensitive:
CREATE SUBSCRIPTION sub3 CONNECTION 'dbname=postgres host=localhost
port=5432' PUBLICATION pub1 with (copy_data= true) CONFLICT RESOLVER
("INSERT_EXISTS" = 'error');
ERROR: INSERT_EXISTS is not a valid conflict type
In few other places it is not case sensitive:
create publication pub1 with ( PUBLISH= 'INSERT,UPDATE,delete');
set log_min_MESSAGES TO warning ;
2.b) Similarly in case of conflict resolver too:
CREATE SUBSCRIPTION sub3 CONNECTION 'dbname=postgres host=localhost
port=5432' PUBLICATION pub1 with (copy_data= true) CONFLICT RESOLVER
("insert_exists" = 'erroR');
ERROR: erroR is not a valid conflict resolver
Fixed.
3) Since there is only one key used to search, we can remove nkeys
variable and directly specify as 1:
+RemoveSubscriptionConflictBySubid(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+ int nkeys = 0;
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict
resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[nkeys++],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, nkeys, skey);
Fixed.
4) Currently we are including CONFLICT RESOLVER even if a subscription
with default CONFLICT RESOLVER is created, we can add the CONFLICT
RESOLVER option only for non-default subscription option:
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ PQExpBuffer InQry = createPQExpBuffer();
+ PGresult *res;
+ int i_confrtype;
+ int i_confrres;
+
+ /* get the conflict types and their resolvers from the
catalog */
+ appendPQExpBuffer(InQry,
+ "SELECT confrtype,
confrres "
+ "FROM
pg_catalog.pg_subscription_conflict"
+ " WHERE confsubid =
%u;\n", subinfo->dobj.catId.oid);
+ res = ExecuteSqlQuery(fout, InQry->data,
PGRES_TUPLES_OK);
+
+ i_confrtype = PQfnumber(res, "confrtype");
+ i_confrres = PQfnumber(res, "confrres");
+
+ if (PQntuples(res) > 0)
+ {
+ appendPQExpBufferStr(query, ") CONFLICT
RESOLVER (");
Fixed.
5) Should remote_apply be apply_remote here as this is what is
specified in code:
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
6) I think this should be "It is the default resolver for
update_origin_differs"
6.a)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>,
<literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
+ It is the default resolver for
<literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
6.b)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-update-differ">
+ <term><literal>update_differ</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
6.c)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>,
<literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
+ It is the default resolver for
<literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
6.d)
+ <varlistentry
id="sql-createsubscription-params-with-conflict_resolver-remote-apply">
+ <term><literal>remote_apply</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>,
<literal>update_exists</literal>,
+ <literal>update_differ</literal> and
<literal>delete_differ</literal>.
Similarly this change should be done in other places too.
Fixed.
7)
7.a) Should delete_differ be changed to delete_origin_differs as that
is what is specified in the subscription commands:
+check_conflict_detection(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and
resolution could be incomplete due to disabled
track_commit_timestamp"),
+ errdetail("Conflicts update_differ and
delete_differ cannot be detected, "
+ "and the origin and
commit timestamp for the local row will not be logged."));
7.b) similarly here too:
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-delete-differ">
+ <term><literal>delete_differ</literal>
(<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was
previously modified
+ by another origin. Note that this conflict can only be
detected when
+ <link
linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is
always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
Similarly this change should be done in other places too.
Fixed.
8) ConflictTypeResolver should be added to typedefs.list to resolve
the pgindent issues:
8.a)
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+
ConflictTypeResolver * resolvers)
Fixed.
8.b) Similarly FormData_pg_subscription_conflict should also be added:
} FormData_pg_subscription_conflict;
/* ----------------
* Form_pg_subscription_conflict corresponds to a pointer to a row with
* the format of pg_subscription_conflict relation.
* ----------------
*/
typedef FormData_pg_subscription_conflict *
Form_pg_subscription_conflict;
Fixed.
On Mon, Sep 9, 2024 at 7:28 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 6, 2024 at 2:05 PM Ajin Cherian <itsajin@gmail.com> wrote:
Thank you for your feedback, Shveta. I've addressed both sets of
comments you provided.
Thanks for the patches. I am reviewing v12-patch001, it is WIP. But
please find first set of comments:1)
src/sgml/logical-replication.sgml:
+ Users have the option to configure a conflict_resolverFull stop for previous line is missing.
Fixed
2)
+ For more information on the conflict_types detected and the
supported conflict_resolvers, refer to section CONFLICT RESOLVERS.We may change to :
For more information on the supported conflict_types and
conflict_resolvers, refer to section CONFLICT RESOLVERS.
Fixed.
3)
src/backend/commands/subscriptioncmds.c:
Line removed. This change is not needed.static void CheckAlterSubOption(Subscription *sub, const char *option,
bool slot_needs_update, bool isTopLevel);
-
Fixed.
4)
Let's stick to the same comments format as the rest of the file i.e.
first letter in caps.+ /* first initialise the resolvers with default values */
first --> First
initialise --> initializeSame for below comments: + /* validate the conflict type and resolver */ + /* update the corresponding resolver for the given conflict type */Please verify the rest of the file for the same.
Fixed.
5)
Please add below in header of parse_subscription_conflict_resolvers
(similar to parse_subscription_options):* This function will report an error if mutually exclusive options
are specified.
Fixed.
6) + * Warn users if prerequisites are not met. + * Initialize with default values. + */ + if (stmt->resolvers) + conf_detection_check_prerequisites(); +Would it be better to move the above call inside
parse_subscription_conflict_resolvers(), then we will have all
resolver related stuff at one place?
Irrespective of whether we move it or not, please remove 'Initialize
with default values.' from above as that is now not done here.
Fixed.
On Mon, Sep 9, 2024 at 7:45 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Thanks for the patches. I tested the v12-0001 patch, and here are my
comments:1) An unexpected error occurs when attempting to alter the resolver
for multiple conflict_type(s) in ALTER SUB...CONFLICT RESOLVER
command. See below examples :postgres=# alter subscription sub2 CONFLICT RESOLVER
(update_exists=keep_local, delete_missing=error,
update_origin_differs=error);
ERROR: unrecognized node type: 1633972341postgres=# alter subscription sub2 CONFLICT RESOLVER (
update_origin_differs=error, update_exists=error);
ERROR: unrecognized node type: 1633972341postgres=# alter subscription sub2 CONFLICT RESOLVER (
delete_origin_differs=error, delete_missing=error);
ERROR: unrecognized node type: 1701602660postgres=# alter subscription sub2 CONFLICT RESOLVER
(update_exists=keep_local, delete_missing=error);
ALTER SUBSCRIPTION-- It appears that the error occurs only when at least two conflict
types belong to the same category, either UPDATE or DELETE.
Fixed this
2) Given the above issue, it would be beneficial to add a test in
subscription.sql to cover cases where all valid conflict types are set
with appropriate resolvers in both the ALTER and CREATE commands.
I've added a few more cases but I feel adding too many tests into "make
check" will make it too long.
I plan to write an alternate script to test this.
Note: As part of this patch the syntax has been changed, now the CONFLICT
RESOLVER comes before the WITH options as suggested by Vignesh.
CREATE SUBSCRIPTION *subscription_name*
CONNECTION '*conninfo*'
PUBLICATION *publication_name* [, ...]
[ CONFLICT RESOLVER ( *conflict_type* [= *conflict_resolver*] [, ...] ) ]
[ WITH ( *subscription_parameter* [= *value*] [, ... ] ) ]
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v13-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v13-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From 57a385df0d0c7752339387594c95d03b3dbde142 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 12 Sep 2024 02:45:33 -0400
Subject: [PATCH v13 3/5] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 44 ++++++++--
src/backend/replication/logical/worker.c | 84 ++++++++++++++++---
src/include/executor/executor.h | 3 +-
src/test/subscription/t/034_conflict_resolver.pl | 100 +++++++++++++++++++++++
4 files changed, 209 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4059707..f944df7 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -566,7 +566,7 @@ static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote)
+ Oid subid, bool apply_remote, ItemPointer tupleid)
{
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
@@ -579,7 +579,7 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
* otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -636,6 +636,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool conflict = false;
ConflictResolver resolver;
bool apply_remote = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -656,11 +657,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
NULL, subid);
- /* Check for conflict and return to caller for resolution if found */
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'insert_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
if (resolver != CR_ERROR &&
has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote))
+ apply_remote, &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -717,7 +723,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -743,6 +750,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -756,6 +765,25 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'update_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, resolver, slot, subid,
+ apply_remote, tid))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fb9a99f..9e9f4b4 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2517,8 +2518,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2670,7 +2672,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2695,14 +2698,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2710,10 +2712,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2754,6 +2757,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2766,7 +2771,32 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3265,6 +3295,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3272,7 +3304,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f390975..291d5dd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 784f89f..d9333dc 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
--
1.8.3.1
v13-0005-Implements-Clock-skew-management-between-nodes.patchapplication/octet-stream; name=v13-0005-Implements-Clock-skew-management-between-nodes.patchDownload
From 545c8bf8c7f395e1772b81acea4e4e89e3bfeb9b Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 12 Sep 2024 04:11:59 -0400
Subject: [PATCH v13 5/5] Implements Clock-skew management between nodes.
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
---
.../replication/logical/applyparallelworker.c | 22 +++-
src/backend/replication/logical/worker.c | 125 ++++++++++++++++++++-
src/backend/utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 +++++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/logicalworker.h | 18 +++
src/include/replication/worker_internal.h | 2 +-
src/include/utils/guc.h | 1 +
src/include/utils/timestamp.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 210 insertions(+), 10 deletions(-)
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 6cbfab0..809dab3 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -314,15 +314,16 @@ pa_can_start(void)
return false;
/*
- * Do not start a new parallel worker if 'last_update_wins' is configured
- * for any conflict type, as we need the commit timestamp in the
- * beginning.
+ * Do not start a new parallel worker if either max clock skew or
+ * 'last_update_wins' is configured for any conflict type. In both of the
+ * cases we need the commit timestamp in the beginning.
*
* XXX: To lift this restriction, we could write the changes to a file
* when a conflict is detected, and then at the commit time, let the
* remaining changes be applied by the apply worker.
*/
- if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ CheckIfSubHasTimeStampResolver(MySubscription->oid))
return false;
return true;
@@ -709,9 +710,20 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured to last_update_wins, thus it is okay to pass 0 as
+ * origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with
+ * last_update_wins resolver, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c2186f8..d6b9e9b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,6 +319,20 @@ static uint32 parallel_stream_nchanges = 0;
bool InitializingApplyWorker = false;
/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
+/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
* Once we start skipping changes, we don't stop it until we skip all changes of
@@ -986,6 +1000,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
}
/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
+/*
* Handle BEGIN message.
*/
static void
@@ -1007,6 +1110,9 @@ apply_handle_begin(StringInfo s)
pgstat_report_activity(STATE_RUNNING, NULL);
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
/*
* Capture the commit timestamp of the remote transaction for time based
* conflict resolution purpose.
@@ -1069,6 +1175,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1324,7 +1433,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2029,7 +2139,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2083,6 +2194,13 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
/*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
+ /*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
*/
@@ -2187,7 +2305,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb404..0ebad6f 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309d..c768a119 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3714,6 +3716,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4992,6 +5021,17 @@ struct config_enum ConfigureNamesEnum[] =
},
{
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
+ {
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc..6424432 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d..7cb0306 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261..95b2a52 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 840b0fe..837a6ec 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -317,6 +317,7 @@ extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
+extern PGDLLIMPORT const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* Functions exported by guc.c
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03e..53b828d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ebb7b9f..bf9e985 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1568,6 +1568,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
1.8.3.1
v13-0004-Implements-last_update_wins-conflict-resolver.patchapplication/octet-stream; name=v13-0004-Implements-last_update_wins-conflict-resolver.patchDownload
From c9d9d19168f00fe0cef1d046bf2d0583e38ef999 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 12 Sep 2024 03:34:45 -0400
Subject: [PATCH v13 4/5] Implements last_update_wins conflict resolver.
This resolver is applicable for conflict types: insert_exists, update_exists,
update_origin_differs and delete_origin_differs.
For these conflicts, when the resolver is set to last_update_wins,
the timestamps of the remote and local conflicting tuple are compared to
determine whether to apply or ignore the remote changes.
The GUC track_commit_timestamp must be enabled to support this resolver.
Since conflict resolution for two phase commit transactions using
prepare-timestamp can result in data divergence, this patch restricts
enabling both two_phase and the last_update_wins resolver together
for a subscription.
The patch also restrict starting a parallel apply worker if resolver is set
to last_update_wins for any conflict type.
---
src/backend/commands/subscriptioncmds.c | 37 +++++-
src/backend/executor/execReplication.c | 68 +++++------
.../replication/logical/applyparallelworker.c | 13 ++
src/backend/replication/logical/conflict.c | 131 +++++++++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 30 +++--
src/bin/pg_dump/pg_dump.c | 38 +++---
src/bin/pg_dump/pg_dump.h | 36 +++---
src/include/replication/conflict.h | 13 +-
src/include/replication/origin.h | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 136 +++++++++++++++++++--
11 files changed, 399 insertions(+), 105 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 131e4f5..6b00cd8 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -450,7 +450,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
*/
static void
parse_subscription_conflict_resolvers(List *stmtresolvers,
- ConflictTypeResolver *resolvers)
+ ConflictTypeResolver *resolvers,
+ bool twophase)
{
ListCell *lc;
List *SeenTypes = NIL;
@@ -484,6 +485,21 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
/* Update the corresponding resolver for the given conflict type. */
resolvers[type].resolver = downcase_truncate_identifier(resolver, strlen(resolver), false);
+
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow enabling both together.
+ *
+ * XXX: An alternative solution idea is that if a conflict is detected
+ * and the resolution strategy is last_update_wins, then start writing
+ * all the changes to a file similar to what we do for streaming mode.
+ * Once commit_prepared arrives, we will read and apply the changes.
+ */
+ if (twophase && pg_strcasecmp(resolvers[type].resolver, "last_update_wins") == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("Setting any resolver to last_update_wins and %s are mutually exclusive options",
+ "two_phase = true")));
}
}
@@ -647,7 +663,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/* Parse and check conflict resolvers. */
- parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers,
+ opts.twophase);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1386,6 +1403,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is
+ * implemented.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable two_phase when a time based resolver is configured")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1643,7 +1673,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
conf_detection_check_prerequisites();
/* Get the list of conflict types and resolvers and validate them. */
- conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers,
+ sub);
/*
* Update the conflict resolvers for the corresponding
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f944df7..5b031a4 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -565,10 +565,20 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
- ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote, ItemPointer tupleid)
+ TupleTableSlot *slot, Oid subid, ItemPointer tupleid)
{
+ ConflictResolver resolver;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+
+ /*
+ * Proceed only if the resolver is not set to 'ERROR'; if the resolver is
+ * 'ERROR', the caller will handle it.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type, NULL, NULL,
+ subid);
+ if (resolver == CR_ERROR)
+ return false;
/* Check all the unique indexes for a conflict */
foreach_oid(uniqueidx, conflictindexes)
@@ -584,8 +594,17 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
+ bool apply_remote = false;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type,
+ &apply_remote, NULL, subid);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
@@ -634,8 +653,6 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
@@ -650,23 +667,10 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'insert_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote, &invalidItemPtr))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, slot, subid,
+ &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -750,8 +754,6 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -765,23 +767,9 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'update_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_UPDATE_EXISTS, resolver, slot, subid,
- apply_remote, tid))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, slot, subid, tid))
return;
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c..6cbfab0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,18 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Do not start a new parallel worker if 'last_update_wins' is configured
+ * for any conflict type, as we need the commit timestamp in the
+ * beginning.
+ *
+ * XXX: To lift this restriction, we could write the changes to a file
+ * when a conflict is detected, and then at the commit time, let the
+ * remaining changes be applied by the apply worker.
+ */
+ if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 3db399c..20107d9 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -51,6 +51,7 @@ static const char *const ConflictTypeNames[] = {
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -76,12 +77,12 @@ static const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -391,6 +392,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -672,6 +678,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -717,6 +732,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
}
/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
+/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
*/
@@ -744,7 +795,7 @@ can_create_full_tuple(Relation localrel,
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
List *
-GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+GetAndValidateSubsConflictResolverList(List *stmtresolvers, Subscription *sub)
{
ListCell *lc;
ConflictTypeResolver *conftyperesolver = NULL;
@@ -779,6 +830,21 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
conftyperesolver->conflict_type,
conftyperesolver->resolver);
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is implemented.
+ */
+ if ((pg_strcasecmp(conftyperesolver->resolver, "last_update_wins") == 0) &&
+ sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver for a subscription that has two_phase enabled",
+ "last_update_wins")));
+
/* Add the conflict type to the list of seen types */
conflictTypes = lappend(conflictTypes,
makeString((char *) conftyperesolver->conflict_type));
@@ -1018,15 +1084,31 @@ RemoveSubscriptionConflictResolvers(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1054,3 +1136,40 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index baf696d..3094030 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -159,6 +159,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9e9f4b4..c2186f8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1006,6 +1006,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -2738,7 +2744,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
{
TupleTableSlot *newslot;
- resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2811,7 +2818,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* the configured resolver is in favor of applying the change, convert
* UPDATE to INSERT and apply the change.
*/
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -2964,7 +2971,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_DELETE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2993,7 +3001,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* The tuple to be deleted could not be found. Based on resolver
* configured, either skip and log a message or emit an error.
*/
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MySubscription->oid);
@@ -3191,7 +3199,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -3234,7 +3242,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
newslot = table_slot_create(partrel, &estate->es_tupleTable);
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, partrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -4766,6 +4775,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4806,10 +4816,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 27f9f30..a5afbe3 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4828,8 +4828,8 @@ getSubscriptions(Archive *fout)
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
PQExpBuffer confQuery;
- PGresult *res;
- PGresult *confRes;
+ PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -5023,50 +5023,50 @@ getSubscriptions(Archive *fout)
ntuples = PQntuples(confRes);
for (j = 0; j < ntuples; j++)
{
- char *confType = PQgetvalue(confRes, j, 0);
- char *confResVal = PQgetvalue(confRes, j, 1);
+ char *confType = PQgetvalue(confRes, j, 0);
+ char *confResVal = PQgetvalue(confRes, j, 1);
if (strcmp(confType, "delete_missing") == 0)
{
subinfo[i].conflict_resolver[CT_DELETE_MISSING].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_DELETE_MISSING].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
else if (strcmp(confType, "delete_origin_differs") == 0)
{
subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
else if (strcmp(confType, "insert_exists") == 0)
{
subinfo[i].conflict_resolver[CT_INSERT_EXISTS].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_INSERT_EXISTS].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
else if (strcmp(confType, "update_exists") == 0)
{
subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
else if (strcmp(confType, "update_missing") == 0)
{
subinfo[i].conflict_resolver[CT_UPDATE_MISSING].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_UPDATE_MISSING].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
else if (strcmp(confType, "update_origin_differs") == 0)
{
subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].resolver =
- pg_strdup(confResVal);
+ pg_strdup(confResVal);
subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].conflict_type =
- pg_strdup(confType);
+ pg_strdup(confType);
}
}
@@ -5323,12 +5323,12 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (fout->remoteVersion >= 180000)
{
ConflictType idx;
- bool first_resolver = true;
+ bool first_resolver = true;
for (idx = 0; idx < CONFLICT_NUM_TYPES; idx++)
{
if (strcmp(subinfo->conflict_resolver[idx].resolver,
- ConflictResolverNames[ConflictTypeDefaultResolvers[idx]]) != 0)
+ ConflictResolverNames[ConflictTypeDefaultResolvers[idx]]) != 0)
{
if (first_resolver)
{
@@ -5338,7 +5338,7 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
first_resolver = false;
}
else
- appendPQExpBuffer(query,", %s = '%s'",
+ appendPQExpBuffer(query, ", %s = '%s'",
subinfo->conflict_resolver[idx].conflict_type,
subinfo->conflict_resolver[idx].resolver);
}
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 13625c6..9822d2e 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -711,7 +711,7 @@ typedef enum ConflictResolver
/* Error out */
CR_ERROR,
-}ConflictResolver;
+} ConflictResolver;
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
@@ -739,27 +739,27 @@ typedef struct ConflictTypeResolver
{
const char *conflict_type;
const char *resolver;
-}ConflictTypeResolver;
+} ConflictTypeResolver;
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
- const char *rolname;
- char *subenabled;
- char *subbinary;
- char *substream;
- char *subtwophasestate;
- char *subdisableonerr;
- char *subpasswordrequired;
- char *subrunasowner;
- char *subconninfo;
- char *subslotname;
- char *subsynccommit;
- char *subpublications;
- char *suborigin;
- char *suboriginremotelsn;
- char *subfailover;
- ConflictTypeResolver conflict_resolver[CONFLICT_NUM_TYPES];
+ const char *rolname;
+ char *subenabled;
+ char *subbinary;
+ char *substream;
+ char *subtwophasestate;
+ char *subdisableonerr;
+ char *subpasswordrequired;
+ char *subrunasowner;
+ char *subconninfo;
+ char *subslotname;
+ char *subsynccommit;
+ char *subpublications;
+ char *suborigin;
+ char *suboriginremotelsn;
+ char *subfailover;
+ ConflictTypeResolver conflict_resolver[CONFLICT_NUM_TYPES];
} SubscriptionInfo;
/*
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 2c9b61c..f9262c5 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -70,6 +71,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -81,7 +85,7 @@ typedef enum ConflictResolver
/* Error out */
CR_ERROR,
-} ConflictResolver;
+} ConflictResolver;
/* Min and max conflict resolver */
#define CR_MIN CR_APPLY_REMOTE
@@ -110,17 +114,20 @@ extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictResolvers(Oid confid);
-extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers,
+ Subscription *sub);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
Oid subid);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
#endif
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9..dcbbbdf 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index d9333dc..6955e91 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -252,6 +280,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -305,16 +363,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -401,10 +491,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -431,6 +525,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
--
1.8.3.1
v13-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v13-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From b22ee73a4e94e723ef831d0c6860b2325a5208f8 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 12 Sep 2024 02:38:04 -0400
Subject: [PATCH v13 2/5] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 68 ++-
src/backend/replication/logical/conflict.c | 235 +++++++--
src/backend/replication/logical/worker.c | 370 +++++++++++----
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 13 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 581 +++++++++++++++++++++++
7 files changed, 1121 insertions(+), 152 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc..4059707 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,14 +550,59 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
+ TupleTableSlot **conflictslot, ConflictType type,
+ ConflictResolver resolver, TupleTableSlot *slot,
+ Oid subid, bool apply_remote)
+{
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
*
@@ -565,7 +610,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -588,6 +634,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -601,6 +649,20 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index c54f6c6..3db399c 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -31,9 +31,10 @@
#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/worker_internal.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
-#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -100,12 +101,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -148,8 +151,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -168,13 +171,22 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
+
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
@@ -183,13 +195,14 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -257,17 +270,24 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -278,13 +298,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -294,47 +315,62 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"),
+ applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -641,6 +677,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
+/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
@@ -650,7 +749,7 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
ListCell *lc;
ConflictTypeResolver *conftyperesolver = NULL;
List *res = NIL;
- List *conflictTypes = NIL;
+ List *conflictTypes = NIL;
foreach(lc, stmtresolvers)
{
@@ -677,8 +776,8 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
* conflict type
*/
validate_conflict_type_and_resolver(
- conftyperesolver->conflict_type,
- conftyperesolver->resolver);
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
/* Add the conflict type to the list of seen types */
conflictTypes = lappend(conflictTypes,
@@ -833,9 +932,9 @@ conf_detection_check_prerequisites(void)
{
if (!track_commit_timestamp)
ereport(WARNING,
- errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
- errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
"detected, and the origin and commit timestamp for the local row "
"will not be logged."));
}
@@ -911,3 +1010,47 @@ RemoveSubscriptionConflictResolvers(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 925dff9..fb9a99f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2704,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2727,47 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2777,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2911,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2928,51 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3021,19 +3106,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3059,6 +3146,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3071,47 +3161,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3123,23 +3251,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3180,12 +3337,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3201,22 +3366,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebf..f390975 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 296270a..2c9b61c 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -96,12 +98,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
@@ -113,5 +117,10 @@ extern ConflictType validate_conflict_type_and_resolver(const char *conflict_typ
extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7..00ade29 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000..784f89f
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,581 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
1.8.3.1
v13-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v13-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From 76d44c31bf8ba88ca4f9bbc3aa9c457b99f7310b Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 12 Sep 2024 02:29:24 -0400
Subject: [PATCH v13 1/5] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for reseting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 88 +-----
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 12 +
doc/src/sgml/ref/create_subscription.sgml | 178 +++++++++++
src/backend/commands/subscriptioncmds.c | 98 ++++++
src/backend/parser/gram.y | 49 ++-
src/backend/replication/logical/conflict.c | 422 +++++++++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 96 +++++-
src/bin/pg_dump/pg_dump.h | 118 ++++++-
src/bin/psql/describe.c | 14 +-
src/bin/psql/tab-complete.c | 6 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_subscription_conflict.h | 55 ++++
src/include/nodes/parsenodes.h | 6 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 51 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 284 ++++++++++++-----
src/test/regress/sql/subscription.sql | 57 ++++
src/tools/pgindent/typedefs.list | 2 +
21 files changed, 1369 insertions(+), 186 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index df62eb4..f82adfc 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,83 +1582,15 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
+ Additional logging is triggered in various conflict scenarios, each identified as a
+ conflict type, and the conflict statistics are collected (displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view).
+ Users have the option to configure a <literal>conflict_resolver</literal> for each
+ <literal>conflict_type</literal> when creating a subscription.
+ For more information on the supported <literal>conflict_types</literal> detected and
+ <literal>conflict_resolvers</literal>, refer to section
+ <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>.
+
Note that there are other conflict scenarios, such as exclusion constraint
violations. Currently, we do not provide additional details for them in the
log.
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 933de6f..6ead5c3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d..e7b39f2
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,7 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
</synopsis>
</refsynopsisdiv>
@@ -345,6 +346,17 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d94..e04a269
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -24,6 +24,7 @@ PostgreSQL documentation
CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceable>
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] ) ]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,183 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc63..131e4f5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -14,6 +14,7 @@
#include "postgres.h"
+#include "access/commit_ts.h"
#include "access/htup_details.h"
#include "access/table.h"
#include "access/twophase.h"
@@ -28,6 +29,7 @@
#include "catalog/pg_database_d.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_subscription_conflict.h"
#include "catalog/pg_type.h"
#include "commands/dbcommands.h"
#include "commands/defrem.h"
@@ -36,7 +38,9 @@
#include "executor/executor.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
+#include "parser/scansup.h"
#include "pgstat.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -440,6 +444,50 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command.
+ * This function will report an error if mutually exclusive or duplicate
+ * options are specified.
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver *resolvers)
+{
+ ListCell *lc;
+ List *SeenTypes = NIL;
+
+ /* Warn users if prerequisites are not met. */
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+ /* First initialize the resolvers with default values. */
+ SetDefaultResolvers(resolvers);
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+ char *resolver;
+
+ /* Check if the conflict type already exists in the list */
+ if (list_member(SeenTypes, makeString(defel->defname)))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+
+ /* Validate the conflict type and resolver. */
+ resolver = defGetString(defel);
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* Add the conflict type to the list of seen types */
+ SeenTypes = lappend(SeenTypes, makeString((char *)resolvers[type].conflict_type));
+
+ /* Update the corresponding resolver for the given conflict type. */
+ resolvers[type].resolver = downcase_truncate_identifier(resolver, strlen(resolver), false);
+ }
+}
+
+/*
* Add publication names from the list to a string.
*/
static void
@@ -583,6 +631,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
/*
* Parse and check options.
@@ -597,6 +646,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /* Parse and check conflict resolvers. */
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -723,6 +775,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1636,46 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ conf_detection_check_prerequisites();
+
+ /* Get the list of conflict types and resolvers and validate them. */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog.
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
+ {
+ ConflictTypeResolver conflictResolvers[CT_MAX + 1];
+
+ /* Remove the existing conflict resolvers. */
+ RemoveSubscriptionConflictResolvers(subid);
+
+ /*
+ * Create list of conflict resolvers and set them in the
+ * catalog.
+ */
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers, CT_MAX + 1);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER:
+ {
+ /*
+ * Reset the conflict resolver for this conflict type to its
+ * default.
+ */
+ ResetConflictResolver(subid, stmt->conflict_type);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1927,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 84cef57..3c090b0 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -612,6 +612,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -769,7 +770,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8754,6 +8755,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10688,14 +10694,15 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_resolver_definition opt_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
n->subname = $3;
n->conninfo = $5;
n->publication = $7;
- n->options = $8;
+ n->resolvers = $8;
+ n->options = $9;
$$ = (Node *) n;
}
;
@@ -10802,6 +10809,38 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
;
/*****************************************************************************
@@ -17725,6 +17764,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18353,6 +18393,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff62..c54f6c6 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,13 +15,28 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heaptoast.h"
+#include "access/heapam.h"
+#include "access/table.h"
+#include "access/tableam.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription.h"
+#include "catalog/pg_subscription_conflict.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "catalog/pg_inherits.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -32,6 +47,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -489,3 +553,361 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ListCell *lc;
+ ConflictTypeResolver *conftyperesolver = NULL;
+ List *res = NIL;
+ List *conflictTypes = NIL;
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ char *resolver_str;
+
+ /* Check if the conflict type already exists in the list */
+ if (list_member(conflictTypes, makeString(defel->defname)))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+ }
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type = downcase_truncate_identifier(defel->defname,
+ strlen(defel->defname), false);
+ resolver_str = defGetString(defel);
+ conftyperesolver->resolver = downcase_truncate_identifier(resolver_str,
+ strlen(resolver_str), false);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+
+ /* Add the conflict type to the list of seen types */
+ conflictTypes = lappend(conflictTypes,
+ makeString((char *) conftyperesolver->conflict_type));
+
+ /* Add the validated ConflictTypeResolver to the result list */
+ res = lappend(res, conftyperesolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *conftyperesolver = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conftyperesolver->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver conflictResolver;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ Relation pg_subscription_conflict;
+ HeapTuple oldtup,
+ newtup;
+ bool valid = false;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Get the index for this conflict_type */
+ for (idx = CT_MIN; idx <= CT_MAX; idx++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[idx], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Get the default resolver for this conflict_type. */
+ conflictResolver.resolver = ConflictResolverNames[ConflictTypeDefaultResolvers[idx]];
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u", conflict_type, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confrres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /* Check if current resolver is the default one, if not update it. */
+ if (pg_strcasecmp(cur_conflict_res, conflictResolver.resolver) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conflictResolver.resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int idx;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+
+ for (idx = 0; idx < resolvers_cnt; idx++)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[idx].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[idx].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[0],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, 1, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 546e7e4..27f9f30 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4827,7 +4827,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
- PGresult *res;
+ PQExpBuffer confQuery;
+ PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4848,7 +4850,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5009,6 +5013,67 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+ for (j = 0; j < ntuples; j++)
+ {
+ char *confType = PQgetvalue(confRes, j, 0);
+ char *confResVal = PQgetvalue(confRes, j, 1);
+
+ if (strcmp(confType, "delete_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "delete_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "insert_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5254,6 +5319,33 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ ConflictType idx;
+ bool first_resolver = true;
+
+ for (idx = 0; idx < CONFLICT_NUM_TYPES; idx++)
+ {
+ if (strcmp(subinfo->conflict_resolver[idx].resolver,
+ ConflictResolverNames[ConflictTypeDefaultResolvers[idx]]) != 0)
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query,", %s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ }
+ }
+
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 0b7d21b..13625c6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -654,24 +654,112 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+#define CONFLICT_NUM_TYPES 6
+
+/*
+ * Conflict types that could occur while applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_ORIGIN_DIFFERS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_ORIGIN_DIFFERS,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /*
+ * Other conflicts, such as exclusion constraint violations, involve more
+ * complex rules than simple equality checks. These conflicts are left for
+ * future improvements.
+ */
+} ConflictType;
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+}ConflictResolver;
+
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+}ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
- const char *rolname;
- char *subenabled;
- char *subbinary;
- char *substream;
- char *subtwophasestate;
- char *subdisableonerr;
- char *subpasswordrequired;
- char *subrunasowner;
- char *subconninfo;
- char *subslotname;
- char *subsynccommit;
- char *subpublications;
- char *suborigin;
- char *suboriginremotelsn;
- char *subfailover;
+ const char *rolname;
+ char *subenabled;
+ char *subbinary;
+ char *substream;
+ char *subtwophasestate;
+ char *subdisableonerr;
+ char *subpasswordrequired;
+ char *subrunasowner;
+ char *subconninfo;
+ char *subslotname;
+ char *subsynccommit;
+ char *subpublications;
+ char *suborigin;
+ char *suboriginremotelsn;
+ char *subfailover;
+ ConflictTypeResolver conflict_resolver[CONFLICT_NUM_TYPES];
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7c9a1f2..c577b3d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6539,7 +6539,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6619,12 +6619,20 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* Add conflict resolvers information from pg_subscription_conflict */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", (SELECT string_agg(confrtype || ' = ' || confrres, ', ') \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " WHERE c.confsubid = s.oid) AS \"%s\"\n",
+ gettext_noop("Conflict Resolvers"));
}
/* Only display subscriptions in current database. */
appendPQExpBufferStr(&buf,
- "FROM pg_catalog.pg_subscription\n"
- "WHERE subdbid = (SELECT oid\n"
+ "FROM pg_catalog.pg_subscription s\n"
+ "WHERE s.subdbid = (SELECT oid\n"
" FROM pg_catalog.pg_database\n"
" WHERE datname = pg_catalog.current_database())");
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index a7ccde6..fa3245e 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -3338,8 +3338,12 @@ psql_completion(const char *text, int start, int end)
{
/* complete with nothing here as this refers to remote publications */
}
+ /* Complete "CREATE SUBSCRIPTION <name> ... CONFLICT RESOLVER ( <confoptions> ) */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("PUBLICATION", MatchAny))
- COMPLETE_WITH("WITH (");
+ COMPLETE_WITH("CONFLICT RESOLVER", "WITH (");
+ else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("CONFLICT", "RESOLVER", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a..f2611c1 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1da..959e1d9 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000..cf35e0d
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict *Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_confsubid_confrtype_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_confsubid_confrtype_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d6f7e79..eddc457 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4197,6 +4197,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4209,6 +4210,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4219,6 +4223,8 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f865907..d661a06 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -375,6 +375,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677..296270a 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -50,6 +50,47 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min and max conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +103,15 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictResolvers(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void ResetConflictResolver(Oid subid, char *conflict_type);
+extern void conf_detection_check_prerequisites(void);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb89..42bf2cc 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..595f496 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,14 +393,146 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- fail - duplicate conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: duplicate conflict type "insert_exists" found
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | keep_local
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: foo is not a valid conflict resolver
+-- fail - altering with duplicate conflict types
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+ERROR: duplicate conflict type "insert_exists" found
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | error
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | keep_local
+ update_missing | skip
+ update_origin_differs | error
+(6 rows)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+------------
+ delete_missing | error
+ delete_origin_differs | keep_local
+ insert_exists | error
+ update_exists | keep_local
+ update_missing | skip
+ update_origin_differs | error
+(6 rows)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
ERROR: disable_on_error requires a Boolean value
@@ -409,18 +541,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..596f2e1 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,63 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+
+-- fail - duplicate conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- fail - altering with duplicate conflict types
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e9ebddd..ebb7b9f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -469,6 +469,7 @@ ConditionalStack
ConfigData
ConfigVariable
ConflictType
+ConflictTypeResolver
ConnCacheEntry
ConnCacheKey
ConnParams
@@ -863,6 +864,7 @@ FormData_pg_statistic
FormData_pg_statistic_ext
FormData_pg_statistic_ext_data
FormData_pg_subscription
+FormData_pg_subscription_conflict
FormData_pg_subscription_rel
FormData_pg_tablespace
FormData_pg_transform
--
1.8.3.1
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type + { + AlterSubscriptionStmt *n = + makeNode(AlterSubscriptionStmt); + + n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER; + n->subname = $3; + n->conflict_type = $8; + $$ = (Node *) n; + } + ; +conflict_type: + Sconst { $$ = $1; } + | NULL_P { $$ = NULL; } ;May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;Fixed.
Few comments:
1) Tab completion missing for:
a) ALTER SUBSCRIPTION name CONFLICT RESOLVER
b) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
c) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR
2) Documentation missing for:
a) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
b) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR
3) This reset is not required here, if valid was false it would have
thrown an error and exited:
a)
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict
type", conflict_type));
+
+ /* Reset */
+ valid = false;
b)
Similarly here too:
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict
resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
4) How about adding CT_MAX inside the enum itself as the last enum value:
typedef enum
{
/* The row to be inserted violates unique constraint */
CT_INSERT_EXISTS,
/* The row to be updated was modified by a different origin */
CT_UPDATE_ORIGIN_DIFFERS,
/* The updated row value violates unique constraint */
CT_UPDATE_EXISTS,
/* The row to be updated is missing */
CT_UPDATE_MISSING,
/* The row to be deleted was modified by a different origin */
CT_DELETE_ORIGIN_DIFFERS,
/* The row to be deleted is missing */
CT_DELETE_MISSING,
/*
* Other conflicts, such as exclusion constraint violations, involve more
* complex rules than simple equality checks. These conflicts are left for
* future improvements.
*/
} ConflictType;
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
/* Min and max conflict type */
#define CT_MIN CT_INSERT_EXISTS
#define CT_MAX CT_DELETE_MISSING
and the for loop can be changed to:
for (type = 0; type < CT_MAX; type++)
This way CT_MIN can be removed and CT_MAX need not be changed every
time a new enum is added.
Also the following +1 can be removed from the variables:
ConflictTypeResolver conflictResolvers[CT_MAX + 1];
5) Similar thing can be done with ConflictResolver enum too. i.e
remove CR_MIN and add CR_MAX as the last element of enum
typedef enum ConflictResolver
{
/* Apply the remote change */
CR_APPLY_REMOTE = 1,
/* Keep the local change */
CR_KEEP_LOCAL,
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
/* Apply the remote change; emit error if it can not be applied */
CR_APPLY_OR_ERROR,
/* Skip applying the change */
CR_SKIP,
/* Error out */
CR_ERROR,
} ConflictResolver;
/* Min and max conflict resolver */
#define CR_MIN CR_APPLY_REMOTE
#define CR_MAX CR_ERROR
6) Except scansup.h inclusion, other inclusions added are not required
in subscriptioncmds.c file.
7)The inclusions "access/heaptoast.h", "access/table.h",
"access/tableam.h", "catalog/dependency.h",
"catalog/pg_subscription.h", "catalog/pg_subscription_conflict.h" and
"catalog/pg_inherits.h" are not required in conflict.c file.
8) Can we change this to use the new foreach_ptr implementations added:
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+ char *resolver;
to use foreach_ptr like:
foreach_ptr(DefElem, defel, stmtresolvers)
{
+ ConflictType type;
+ char *resolver;
....
}
Regards,
Vignesh
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type + { + AlterSubscriptionStmt *n = + makeNode(AlterSubscriptionStmt); + + n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER; + n->subname = $3; + n->conflict_type = $8; + $$ = (Node *) n; + } + ; +conflict_type: + Sconst { $$ = $1; } + | NULL_P { $$ = NULL; } ;May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;Fixed.
Few comments:
1) This should be in (fout->remoteVersion >= 180000) check to support
dumping backward compatible server objects, else dump with older
version will fail:
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT confrtype,
confrres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid =
%u;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data,
PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+ for (j = 0; j < ntuples; j++)
2) Can we check and throw an error before the warning is logged in
this case as it seems strange to throw a warning first and then an
error for the same track_commit_timestamp configuration:
postgres=# create subscription sub1 connection ... publication pub1
conflict resolver (insert_exists = 'last_update_wins');
WARNING: conflict detection and resolution could be incomplete due to
disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs
cannot be detected, and the origin and commit timestamp for the local
row will not be logged.
ERROR: resolver last_update_wins requires "track_commit_timestamp" to
be enabled
HINT: Make sure the configuration parameter "track_commit_timestamp" is set.
Regards,
Vignesh
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
I was reviewing the CONFLICT RESOLVER (insert_exists='apply_remote')
and found that one conflict remains unresolved in the following
scenario:
Pub:
CREATE TABLE circles(c1 CIRCLE, c2 text, EXCLUDE USING gist (c1 WITH &&));
CREATE PUBLICATION pub1 for table circles;
Sub:
CREATE TABLE circles(c1 CIRCLE, c2 text, EXCLUDE USING gist (c1 WITH &&))
insert into circles values('<(0,0), 5>', 'sub');
CREATE SUBSCRIPTION ... PUBLICATION pub1 CONFLICT RESOLVER
(insert_exists='apply_remote');
The following conflict is not detected and resolved with remote tuple data:
Pub:
INSERT INTO circles VALUES('<(0,0), 5>', 'pub');
2024-09-19 17:32:36.637 IST [31463] 31463 LOG: conflict detected on
relation "public.t1": conflict=insert_exists, Resolution=apply_remote.
2024-09-19 17:32:36.637 IST [31463] 31463 DETAIL: Key already
exists in unique index "t1_pkey", modified in transaction 742,
applying the remote changes.
Key (c1)=(1); existing local tuple (1, sub); remote tuple (1, pub).
2024-09-19 17:32:36.637 IST [31463] 31463 CONTEXT: processing
remote data for replication origin "pg_16398" during message type
"INSERT" for replication target relation "public.t1" in transaction
744, finished at 0/1528E88
........
2024-09-19 17:32:44.653 IST [31463] 31463 ERROR: conflicting key
value violates exclusion constraint "circles_c1_excl"
2024-09-19 17:32:44.653 IST [31463] 31463 DETAIL: Key
(c1)=(<(0,0),5>) conflicts with existing key (c1)=(<(0,0),5>).
........
Regards,
Vignesh
Hello!
Sorry for being noisy, just for the case, want to notice that [1]/messages/by-id/OS0PR01MB5716E30952F542E256DD72E294802@OS0PR01MB5716.jpnprd01.prod.outlook.com needs to
be addressed before any real usage of conflict resolution.
[1]: /messages/by-id/OS0PR01MB5716E30952F542E256DD72E294802@OS0PR01MB5716.jpnprd01.prod.outlook.com
/messages/by-id/OS0PR01MB5716E30952F542E256DD72E294802@OS0PR01MB5716.jpnprd01.prod.outlook.com
On Wed, Sep 18, 2024 at 10:46 AM vignesh C <vignesh21@gmail.com> wrote:
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type + { + AlterSubscriptionStmt *n = + makeNode(AlterSubscriptionStmt); + + n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER; + n->subname = $3; + n->conflict_type = $8; + $$ = (Node *) n; + } + ; +conflict_type: + Sconst { $$ = $1; } + | NULL_P { $$ = NULL; } ;May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;Fixed.
Few comments: 1) This should be in (fout->remoteVersion >= 180000) check to support dumping backward compatible server objects, else dump with older version will fail: + /* Populate conflict type fields using the new query */ + confQuery = createPQExpBuffer(); + appendPQExpBuffer(confQuery, + "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict " + "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid); + confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK); + + ntuples = PQntuples(confRes); + for (j = 0; j < ntuples; j++)2) Can we check and throw an error before the warning is logged in
this case as it seems strange to throw a warning first and then an
error for the same track_commit_timestamp configuration:
postgres=# create subscription sub1 connection ... publication pub1
conflict resolver (insert_exists = 'last_update_wins');
WARNING: conflict detection and resolution could be incomplete due to
disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs
cannot be detected, and the origin and commit timestamp for the local
row will not be logged.
ERROR: resolver last_update_wins requires "track_commit_timestamp" to
be enabled
HINT: Make sure the configuration parameter "track_commit_timestamp" is set.
Thanks for the review.
Here is the v14 patch-set fixing review comments in [1]/messages/by-id/CALDaNm3es1JqU8Qcv5Yw=7Ts2dOvaV8a_boxPSdofB+DTx1oFg@mail.gmail.com and [2]/messages/by-id/CALDaNm18HuAcNsEC47J6qLRC7rMD2Q9_wT_hFtcc4UWqsfkgjA@mail.gmail.com.
New in patches:
1) Added partition table tests in 034_conflict_resolver.pl in 002 and
003 patches.
2) 003 has a bug fix for update_exists conflict resolution on
partitioned tables.
[1]: /messages/by-id/CALDaNm3es1JqU8Qcv5Yw=7Ts2dOvaV8a_boxPSdofB+DTx1oFg@mail.gmail.com
[2]: /messages/by-id/CALDaNm18HuAcNsEC47J6qLRC7rMD2Q9_wT_hFtcc4UWqsfkgjA@mail.gmail.com
Thanks,
Nisha
Attachments:
v14-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v14-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From e02fed3179fa21cd612ec79f873a43b3c0136453 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Thu, 19 Sep 2024 06:24:27 -0400
Subject: [PATCH v14 1/5] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for reseting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 88 +---
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 26 +-
doc/src/sgml/ref/create_subscription.sgml | 178 ++++++++
src/backend/commands/subscriptioncmds.c | 99 +++++
src/backend/parser/gram.y | 49 ++-
src/backend/replication/logical/conflict.c | 413 ++++++++++++++++++
src/bin/pg_dump/pg_dump.c | 96 +++-
src/bin/pg_dump/pg_dump.h | 118 ++++-
src/bin/psql/describe.c | 14 +-
src/bin/psql/tab-complete.c | 22 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 55 +++
src/include/nodes/parsenodes.h | 6 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 51 +++
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 270 ++++++++----
src/test/regress/sql/subscription.sql | 57 +++
src/tools/pgindent/typedefs.list | 2 +
21 files changed, 1375 insertions(+), 188 deletions(-)
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index df62eb45ff..f82adfc4ed 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,83 +1582,15 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
+ Additional logging is triggered in various conflict scenarios, each identified as a
+ conflict type, and the conflict statistics are collected (displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view).
+ Users have the option to configure a <literal>conflict_resolver</literal> for each
+ <literal>conflict_type</literal> when creating a subscription.
+ For more information on the supported <literal>conflict_types</literal> detected and
+ <literal>conflict_resolvers</literal>, refer to section
+ <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>.
+
Note that there are other conflict scenarios, such as exclusion constraint
violations. Currently, we do not provide additional details for them in the
log.
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a2fda4677d..b87c660032 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
old mode 100644
new mode 100755
index fdc648d007..55eae8b875
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ...] )
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER ALL
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER FOR (<replaceable class="parameter">conflict_type</replaceable>)
</synopsis>
</refsynopsisdiv>
@@ -345,7 +348,28 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
- </variablelist>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type being reset to its default resolver setting.
+ For details on conflict types and their default resolvers, refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
<para>
When specifying a parameter of type <type>boolean</type>, the
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
old mode 100644
new mode 100755
index 740b7d9421..25d4c0bb39
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -24,6 +24,7 @@ PostgreSQL documentation
CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceable>
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] ) ]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -432,6 +433,183 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies options for conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist></para>
+
+ <para>
+ The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist></para>
+
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc636b8..859bf086cd 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -36,6 +36,7 @@
#include "executor/executor.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -439,6 +440,52 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
}
}
+/*
+ * Parsing function for conflict resolvers in CREATE SUBSCRIPTION command.
+ * This function will report an error if mutually exclusive or duplicate
+ * options are specified.
+ */
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver *resolvers)
+{
+ ListCell *lc;
+ List *SeenTypes = NIL;
+
+
+ /* First initialize the resolvers with default values. */
+ SetDefaultResolvers(resolvers);
+
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
+ ConflictType type;
+ char *resolver;
+
+ /* Check if the conflict type already exists in the list */
+ if (list_member(SeenTypes, makeString(defel->defname)))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+
+ /* Validate the conflict type and resolver. */
+ resolver = defGetString(defel);
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
+
+ /* Add the conflict type to the list of seen types */
+ SeenTypes = lappend(SeenTypes, makeString((char *)resolvers[type].conflict_type));
+
+ /* Update the corresponding resolver for the given conflict type. */
+ resolvers[type].resolver = downcase_truncate_identifier(resolver, strlen(resolver), false);
+ }
+
+ /* Once validation is complete, warn users if prerequisites are not met. */
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+}
+
/*
* Add publication names from the list to a string.
*/
@@ -583,6 +630,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
/*
* Parse and check options.
@@ -597,6 +645,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /* Parse and check conflict resolvers. */
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -723,6 +774,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolver(subid, conflictResolvers, CONFLICT_NUM_TYPES);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1635,48 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+
+ /* Get the list of conflict types and resolvers and validate them. */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+
+ /* Warn users if prerequisites are not met */
+ conf_detection_check_prerequisites();
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog.
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
+ {
+ ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
+
+ /* Remove the existing conflict resolvers. */
+ RemoveSubscriptionConflictResolvers(subid);
+
+ /*
+ * Create list of conflict resolvers and set them in the
+ * catalog.
+ */
+ SetDefaultResolvers(conflictResolvers);
+ SetSubConflictResolver(subid, conflictResolvers, CONFLICT_NUM_TYPES);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER:
+ {
+ /*
+ * Reset the conflict resolver for this conflict type to its
+ * default.
+ */
+ ResetConflictResolver(subid, stmt->conflict_type);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1928,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubscriptionConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ab304ca989..c4d389721d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -770,7 +771,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8789,6 +8790,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10723,14 +10729,15 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_resolver_definition opt_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
n->subname = $3;
n->conninfo = $5;
n->publication = $7;
- n->options = $8;
+ n->resolvers = $8;
+ n->options = $9;
$$ = (Node *) n;
}
;
@@ -10837,6 +10844,38 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
;
/*****************************************************************************
@@ -17761,6 +17800,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18390,6 +18430,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff626bd..ff44c3c017 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -15,13 +15,21 @@
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heapam.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -32,6 +40,55 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -489,3 +546,359 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+ int i;
+
+ /* Check conflict type validity */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Reset */
+ valid = false;
+
+ /* Check conflict resolver validity. */
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ {
+ if (ConflictTypeResolverMap[type][i] == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+ return type;
+
+}
+
+/*
+ * Extract the conflict type and conflict resolvers from the
+ * ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
+ */
+List *
+GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+{
+ ConflictTypeResolver *conftyperesolver = NULL;
+ List *res = NIL;
+ List *conflictTypes = NIL;
+
+ foreach_ptr(DefElem, defel, stmtresolvers)
+ {
+ char *resolver_str;
+
+ /* Check if the conflict type already exists in the list */
+ if (list_member(conflictTypes, makeString(defel->defname)))
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+ }
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type = downcase_truncate_identifier(defel->defname,
+ strlen(defel->defname), false);
+ resolver_str = defGetString(defel);
+ conftyperesolver->resolver = downcase_truncate_identifier(resolver_str,
+ strlen(resolver_str), false);
+
+ /*
+ * Validate the conflict type and that the resolver is valid for that
+ * conflict type
+ */
+ validate_conflict_type_and_resolver(
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+
+ /* Add the conflict type to the list of seen types */
+ conflictTypes = lappend(conflictTypes,
+ makeString((char *) conftyperesolver->conflict_type));
+
+ /* Add the validated ConflictTypeResolver to the result list */
+ res = lappend(res, conftyperesolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ ListCell *lc;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach(lc, conflict_resolvers)
+ {
+ ConflictTypeResolver *conftyperesolver = (ConflictTypeResolver *) lfirst(lc);
+
+ /* set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conftyperesolver->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type, subid);
+
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver conflictResolver;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ Relation pg_subscription_conflict;
+ HeapTuple oldtup,
+ newtup;
+ bool valid = false;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Get the index for this conflict_type */
+ for (idx = CT_MIN; idx <= CT_MAX; idx++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[idx], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ /* Get the default resolver for this conflict_type. */
+ conflictResolver.resolver = ConflictResolverNames[ConflictTypeDefaultResolvers[idx]];
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_confrtype - 1] = CStringGetTextDatum(conflict_type);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_confrtype - 1]);
+
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u", conflict_type, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confrres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /* Check if current resolver is the default one, if not update it. */
+ if (pg_strcasecmp(cur_conflict_res, conflictResolver.resolver) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conflictResolver.resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ int idx;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple. */
+ memset(nulls, false, sizeof(nulls));
+
+ for (idx = 0; idx < resolvers_cnt; idx++)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_confrtype - 1] =
+ CStringGetTextDatum(resolvers[idx].conflict_type);
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(resolvers[idx].resolver);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubscriptionConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid, this should return all conflict resolvers for
+ * this sub
+ */
+ ScanKeyInit(&skey[0],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, 1, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 130b80775d..5b541324a4 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4827,7 +4827,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
- PGresult *res;
+ PQExpBuffer confQuery;
+ PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4848,7 +4850,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5009,6 +5013,67 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+ for (j = 0; j < ntuples; j++)
+ {
+ char *confType = PQgetvalue(confRes, j, 0);
+ char *confResVal = PQgetvalue(confRes, j, 1);
+
+ if (strcmp(confType, "delete_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "delete_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_DELETE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "insert_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_INSERT_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_exists") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_EXISTS].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_missing") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_MISSING].conflict_type =
+ pg_strdup(confType);
+ }
+ else if (strcmp(confType, "update_origin_differs") == 0)
+ {
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].resolver =
+ pg_strdup(confResVal);
+ subinfo[i].conflict_resolver[CT_UPDATE_ORIGIN_DIFFERS].conflict_type =
+ pg_strdup(confType);
+ }
+
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5254,6 +5319,33 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ ConflictType idx;
+ bool first_resolver = true;
+
+ for (idx = 0; idx < CONFLICT_NUM_TYPES; idx++)
+ {
+ if (strcmp(subinfo->conflict_resolver[idx].resolver,
+ ConflictResolverNames[ConflictTypeDefaultResolvers[idx]]) != 0)
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query,", %s = '%s'",
+ subinfo->conflict_resolver[idx].conflict_type,
+ subinfo->conflict_resolver[idx].resolver);
+ }
+ }
+
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 9f907ed5ad..da9be05a71 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,24 +655,112 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+#define CONFLICT_NUM_TYPES 6
+
+/*
+ * Conflict types that could occur while applying remote changes.
+ */
+typedef enum
+{
+ /* The row to be inserted violates unique constraint */
+ CT_INSERT_EXISTS,
+
+ /* The row to be updated was modified by a different origin */
+ CT_UPDATE_ORIGIN_DIFFERS,
+
+ /* The updated row value violates unique constraint */
+ CT_UPDATE_EXISTS,
+
+ /* The row to be updated is missing */
+ CT_UPDATE_MISSING,
+
+ /* The row to be deleted was modified by a different origin */
+ CT_DELETE_ORIGIN_DIFFERS,
+
+ /* The row to be deleted is missing */
+ CT_DELETE_MISSING,
+
+ /*
+ * Other conflicts, such as exclusion constraint violations, involve more
+ * complex rules than simple equality checks. These conflicts are left for
+ * future improvements.
+ */
+} ConflictType;
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflcit.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+}ConflictResolver;
+
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+}ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
- const char *rolname;
- char *subenabled;
- char *subbinary;
- char *substream;
- char *subtwophasestate;
- char *subdisableonerr;
- char *subpasswordrequired;
- char *subrunasowner;
- char *subconninfo;
- char *subslotname;
- char *subsynccommit;
- char *subpublications;
- char *suborigin;
- char *suboriginremotelsn;
- char *subfailover;
+ const char *rolname;
+ char *subenabled;
+ char *subbinary;
+ char *substream;
+ char *subtwophasestate;
+ char *subdisableonerr;
+ char *subpasswordrequired;
+ char *subrunasowner;
+ char *subconninfo;
+ char *subslotname;
+ char *subsynccommit;
+ char *subpublications;
+ char *suborigin;
+ char *suboriginremotelsn;
+ char *subfailover;
+ ConflictTypeResolver conflict_resolver[CONFLICT_NUM_TYPES];
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index faabecbc76..5a999c4460 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6547,7 +6547,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6627,12 +6627,20 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* Add conflict resolvers information from pg_subscription_conflict */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", (SELECT string_agg(confrtype || ' = ' || confrres, ', ') \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " WHERE c.confsubid = s.oid) AS \"%s\"\n",
+ gettext_noop("Conflict Resolvers"));
}
/* Only display subscriptions in current database. */
appendPQExpBufferStr(&buf,
- "FROM pg_catalog.pg_subscription\n"
- "WHERE subdbid = (SELECT oid\n"
+ "FROM pg_catalog.pg_subscription s\n"
+ "WHERE s.subdbid = (SELECT oid\n"
" FROM pg_catalog.pg_database\n"
" WHERE datname = pg_catalog.current_database())");
diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c
index a7ccde6d7d..8ebdf10b19 100644
--- a/src/bin/psql/tab-complete.c
+++ b/src/bin/psql/tab-complete.c
@@ -1932,7 +1932,8 @@ psql_completion(const char *text, int start, int end)
else if (Matches("ALTER", "SUBSCRIPTION", MatchAny))
COMPLETE_WITH("CONNECTION", "ENABLE", "DISABLE", "OWNER TO",
"RENAME TO", "REFRESH PUBLICATION", "SET", "SKIP (",
- "ADD PUBLICATION", "DROP PUBLICATION");
+ "ADD PUBLICATION", "DROP PUBLICATION", "CONFLICT RESOLVER (",
+ "RESET CONFLICT RESOLVER");
/* ALTER SUBSCRIPTION <name> REFRESH PUBLICATION */
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("REFRESH", "PUBLICATION"))
@@ -1965,6 +1966,19 @@ psql_completion(const char *text, int start, int end)
else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
TailMatches("ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
+ /* ALTER SUBSCRIPTION <name> CONFLICT RESOLVER (<conflict_type> = <conflict_resolver> [, ..]) */
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESOLVER", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
+ /* ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER ALL|FOR <conflict_type> */
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESET", "CONFLICT", "RESOLVER"))
+ COMPLETE_WITH("ALL", "FOR (");
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESOLVER", "FOR", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
@@ -3338,8 +3352,12 @@ psql_completion(const char *text, int start, int end)
{
/* complete with nothing here as this refers to remote publications */
}
+ /* Complete "CREATE SUBSCRIPTION <name> ... CONFLICT RESOLVER ( <confoptions> ) */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("PUBLICATION", MatchAny))
- COMPLETE_WITH("WITH (");
+ COMPLETE_WITH("CONFLICT RESOLVER", "WITH (");
+ else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("CONFLICT", "RESOLVER", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..cf35e0d19b
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict *Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_confsubid_confrtype_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, confrtype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_confsubid_confrtype_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index e62ce1b753..7e19e0c644 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4201,6 +4201,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4213,6 +4214,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4223,6 +4227,8 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 899d64ad55..e62b0c6945 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -376,6 +376,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677ff5..fcd49da9d7 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -50,6 +50,47 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +103,15 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictResolvers(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void ResetConflictResolver(Oid subid, char *conflict_type);
+extern void conf_detection_check_prerequisites(void);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..9fa2a3fe13 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,12 +393,130 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+ERROR: foo is not a valid conflict type
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- fail - duplicate conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+ERROR: duplicate conflict type "insert_exists" found
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | keep_local
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- fail - altering with duplicate conflict types
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+ERROR: duplicate conflict type "insert_exists" found
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | error
+ delete_origin_differs | keep_local
+ insert_exists | apply_remote
+ update_exists | keep_local
+ update_missing | skip
+ update_origin_differs | error
+(6 rows)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+------------
+ delete_missing | error
+ delete_origin_differs | keep_local
+ insert_exists | error
+ update_exists | keep_local
+ update_missing | skip
+ update_origin_differs | error
+(6 rows)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+ confrtype | confrres
+-----------------------+--------------
+ delete_missing | skip
+ delete_origin_differs | apply_remote
+ insert_exists | error
+ update_exists | error
+ update_missing | skip
+ update_origin_differs | apply_remote
+(6 rows)
+
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
-- fail - disable_on_error must be boolean
@@ -409,18 +527,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | insert_exists = error, update_origin_differs = apply_remote, update_exists = error, update_missing = skip, delete_origin_differs = apply_remote, delete_missing = skip
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..596f2e11fd 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,63 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+
+-- fail - invalid conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+
+-- fail - duplicate conflict types
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+
+-- creating subscription should create default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict type and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- fail - altering with duplicate conflict types
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b6135f0347..bbf3221075 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -469,6 +469,7 @@ ConditionalStack
ConfigData
ConfigVariable
ConflictType
+ConflictTypeResolver
ConnCacheEntry
ConnCacheKey
ConnParams
@@ -863,6 +864,7 @@ FormData_pg_statistic
FormData_pg_statistic_ext
FormData_pg_statistic_ext_data
FormData_pg_subscription
+FormData_pg_subscription_conflict
FormData_pg_subscription_rel
FormData_pg_tablespace
FormData_pg_transform
--
2.34.1
v14-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v14-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 70492a11c5e1b52c9052e66e2d38b12aabd7a2b2 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 19 Sep 2024 18:36:30 +0530
Subject: [PATCH v14 2/5] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 68 +-
src/backend/replication/logical/conflict.c | 251 +++--
src/backend/replication/logical/worker.c | 370 ++++++--
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 21 +-
src/test/subscription/meson.build | 1 +
.../subscription/t/034_conflict_resolver.pl | 873 ++++++++++++++++++
7 files changed, 1425 insertions(+), 164 deletions(-)
create mode 100755 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 1086cbc962..4059707a27 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,13 +550,58 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
+/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
+ TupleTableSlot **conflictslot, ConflictType type,
+ ConflictResolver resolver, TupleTableSlot *slot,
+ Oid subid, bool apply_remote)
+{
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -565,7 +610,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -588,6 +634,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -601,6 +649,20 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index ff44c3c017..82a3bc26f4 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -24,9 +24,10 @@
#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "replication/logicalproto.h"
+#include "replication/worker_internal.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
-#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -93,12 +94,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -141,8 +144,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -161,13 +164,22 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
+
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
@@ -176,13 +188,14 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -250,17 +263,24 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -271,13 +291,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -287,47 +308,62 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update."));
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, UPDATE can not be converted to INSERT, hence ERROR out."));
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, Convert UPDATE to INSERT and %s"),
+ applymsg);
+ else
+ appendStringInfo(&err_detail, _("Could not find the row to be updated, %s"),
+ applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -552,7 +588,7 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
* Set default values for CONFLICT RESOLVERS for each conflict type
*/
void
-SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+SetDefaultResolvers(ConflictTypeResolver *conflictResolvers)
{
ConflictType type;
@@ -633,6 +669,69 @@ validate_conflict_type_and_resolver(const char *conflict_type,
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ tuple, Anum_pg_subscription_conflict_confrres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Extract the conflict type and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
@@ -642,11 +741,11 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
{
ConflictTypeResolver *conftyperesolver = NULL;
List *res = NIL;
- List *conflictTypes = NIL;
+ List *conflictTypes = NIL;
foreach_ptr(DefElem, defel, stmtresolvers)
{
- char *resolver_str;
+ char *resolver_str;
/* Check if the conflict type already exists in the list */
if (list_member(conflictTypes, makeString(defel->defname)))
@@ -658,18 +757,18 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
conftyperesolver = palloc(sizeof(ConflictTypeResolver));
conftyperesolver->conflict_type = downcase_truncate_identifier(defel->defname,
- strlen(defel->defname), false);
+ strlen(defel->defname), false);
resolver_str = defGetString(defel);
conftyperesolver->resolver = downcase_truncate_identifier(resolver_str,
- strlen(resolver_str), false);
+ strlen(resolver_str), false);
/*
* Validate the conflict type and that the resolver is valid for that
* conflict type
*/
validate_conflict_type_and_resolver(
- conftyperesolver->conflict_type,
- conftyperesolver->resolver);
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
/* Add the conflict type to the list of seen types */
conflictTypes = lappend(conflictTypes,
@@ -721,7 +820,7 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
{
/* Update the new resolver */
values[Anum_pg_subscription_conflict_confrres - 1] =
- CStringGetTextDatum(conftyperesolver->resolver);
+ CStringGetTextDatum(conftyperesolver->resolver);
replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
@@ -824,18 +923,18 @@ conf_detection_check_prerequisites(void)
{
if (!track_commit_timestamp)
ereport(WARNING,
- errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
- errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
- errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
- "detected, and the origin and commit timestamp for the local row "
- "will not be logged."));
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
}
/*
* Set Conflict Resolvers on the subscription
*/
void
-SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+SetSubConflictResolver(Oid subId, ConflictTypeResolver *resolvers, int resolvers_cnt)
{
Relation pg_subscription_conflict;
Datum values[Natts_pg_subscription_conflict];
@@ -902,3 +1001,47 @@ RemoveSubscriptionConflictResolvers(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver of the conflict type set under the given subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 925dff9cc4..fb9a99f9a8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,36 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2704,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2727,47 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2777,28 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ if (apply_remote)
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
}
/* Cleanup. */
@@ -2848,6 +2911,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2928,51 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ &apply_remote, NULL,
+ MySubscription->oid);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3021,19 +3106,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3059,6 +3146,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3071,47 +3161,85 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ &apply_remote, newtup,
+ MySubscription->oid);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ &apply_remote, NULL,
+ MySubscription->oid);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3123,23 +3251,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3180,12 +3337,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3201,22 +3366,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00..f3909759f8 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index fcd49da9d7..c68bf6fe7e 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -10,6 +10,8 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -79,7 +81,7 @@ typedef enum ConflictResolver
/* Error out */
CR_ERROR,
-} ConflictResolver;
+} ConflictResolver;
/* Min conflict resolver */
#define CR_MIN CR_APPLY_REMOTE
@@ -89,29 +91,36 @@ typedef struct ConflictTypeResolver
{
const char *conflict_type;
const char *resolver;
-} ConflictTypeResolver;
+} ConflictTypeResolver;
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver *resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictResolvers(Oid confid);
extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
-extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void SetDefaultResolvers(ConflictTypeResolver *conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
+extern ConflictResolver GetConflictResolver(Relation localrel,
+ ConflictType type,
+ bool *apply_remote,
+ LogicalRepTupleData *newtup,
+ Oid subid);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100755
index 0000000000..0091443224
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,873 @@
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp_set
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+#test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, UPDATE can not be converted to INSERT, hence SKIP the update./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# test the apply part
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+#test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+#################################################
+# Partition table tests for UPDATE conflicts
+#################################################
+
+# Create partitioned table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);"
+);
+
+# Create similar table on subscriber but with partitions
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text) partition by range (b);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);
+ CREATE TABLE conf_tab_part_1 PARTITION OF conf_tab_part FOR VALUES FROM (MINVALUE) TO (100);
+ CREATE TABLE conf_tab_part_2 PARTITION OF conf_tab_part FOR VALUES FROM (101) TO (MAXVALUE);"
+);
+
+# Setup logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_part FOR TABLE conf_tab_part with (publish_via_partition_root=true);"
+);
+
+# Create the subscription
+$appname = 'sub_part';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=1);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is applied to the first partition");
+
+# Create a conflicting update which also changes the partition
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=101, data='frompubnew_p2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=1);");
+
+is($result, qq(101|frompubnew_p2),
+ "update from remote is the second partition");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept on the first partition");
+
+# Create a conflicting update which also changes the partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=2);");
+
+is($result, qq(1|fromsub),
+ "update from local is kept on the first partition");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs./,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+#Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=3);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is converted to insert in the first partition");
+
+# Test the update which also changes the partition
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=103, data='frompubnew_p2' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b,data from conf_tab_part WHERE (a=3);");
+
+is($result, '103|frompubnew_p2',
+ "update from remote is converted to insert in the second partition");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on first partition is skipped on the subscriber");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on second partition is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+#Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v14-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v14-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From 4bab5951866751a7e834628e5256f651d09dd92e Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 19 Sep 2024 18:06:41 +0530
Subject: [PATCH v14 3/5] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 44 +++-
src/backend/replication/logical/worker.c | 84 ++++++--
src/include/executor/executor.h | 3 +-
.../subscription/t/034_conflict_resolver.pl | 201 ++++++++++++++++++
4 files changed, 310 insertions(+), 22 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 4059707a27..f944df761c 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -566,7 +566,7 @@ static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote)
+ Oid subid, bool apply_remote, ItemPointer tupleid)
{
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
@@ -579,7 +579,7 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
* otherwise return to caller for resolutions.
*/
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -636,6 +636,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
bool conflict = false;
ConflictResolver resolver;
bool apply_remote = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -656,11 +657,16 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
NULL, subid);
- /* Check for conflict and return to caller for resolution if found */
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'insert_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
if (resolver != CR_ERROR &&
has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote))
+ apply_remote, &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -717,7 +723,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -743,6 +750,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
+ ConflictResolver resolver;
+ bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -756,6 +765,25 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Get the configured resolver and determine if remote changes should
+ * be applied.
+ */
+ resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
+ NULL, subid);
+
+ /*
+ * Check for conflict and return to caller for resolution if found.
+ * Only perform this check if the 'update_exists' resolver is not set
+ * to 'ERROR'; if the resolver is 'ERROR', the upcoming
+ * CheckAndReportConflict() call will handle it.
+ */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, resolver, slot, subid,
+ apply_remote, tid))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fb9a99f9a8..bfb9c37071 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2517,8 +2518,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2670,7 +2672,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2695,14 +2698,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2710,10 +2712,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2754,6 +2757,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2766,7 +2771,32 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3265,6 +3295,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3272,7 +3304,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f3909759f8..291d5dde36 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 0091443224..9a9c75b835 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -870,4 +970,105 @@ $node_subscriber->wait_for_log(
qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing/,
$log_offset);
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (2,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (3,1,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (4,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (5,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (6,1,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=1);");
+
+is($result, 'frompub', "update from remote on partition is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from remote on partition is skipped");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists/,
+ $log_offset);
+
done_testing();
--
2.34.1
v14-0004-Implements-last_update_wins-conflict-resolver.patchapplication/octet-stream; name=v14-0004-Implements-last_update_wins-conflict-resolver.patchDownload
From d491bced4c2b1199dfe3d202df5c53d77d1fab34 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 19 Sep 2024 19:12:51 +0530
Subject: [PATCH v14 4/5] Implements last_update_wins conflict resolver.
This resolver is applicable for conflict types: insert_exists, update_exists,
update_origin_differs and delete_origin_differs.
For these conflicts, when the resolver is set to last_update_wins,
the timestamps of the remote and local conflicting tuple are compared to
determine whether to apply or ignore the remote changes.
The GUC track_commit_timestamp must be enabled to support this resolver.
Since conflict resolution for two phase commit transactions using
prepare-timestamp can result in data divergence, this patch restricts
enabling both two_phase and the last_update_wins resolver together
for a subscription.
The patch also restrict starting a parallel apply worker if resolver is set
to last_update_wins for any conflict type.
---
src/backend/commands/subscriptioncmds.c | 44 +++++-
src/backend/executor/execReplication.c | 68 ++++-----
.../replication/logical/applyparallelworker.c | 13 ++
src/backend/replication/logical/conflict.c | 131 ++++++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 30 ++--
src/include/replication/conflict.h | 11 +-
src/include/replication/origin.h | 1 +
.../subscription/t/034_conflict_resolver.pl | 136 +++++++++++++++++-
9 files changed, 366 insertions(+), 69 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 859bf086cd..21ae8c6c6f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -447,7 +447,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options,
*/
static void
parse_subscription_conflict_resolvers(List *stmtresolvers,
- ConflictTypeResolver *resolvers)
+ ConflictTypeResolver *resolvers,
+ bool twophase)
{
ListCell *lc;
List *SeenTypes = NIL;
@@ -474,10 +475,25 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
defGetString(defel));
/* Add the conflict type to the list of seen types */
- SeenTypes = lappend(SeenTypes, makeString((char *)resolvers[type].conflict_type));
+ SeenTypes = lappend(SeenTypes, makeString((char *) resolvers[type].conflict_type));
/* Update the corresponding resolver for the given conflict type. */
resolvers[type].resolver = downcase_truncate_identifier(resolver, strlen(resolver), false);
+
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow enabling both together.
+ *
+ * XXX: An alternative solution idea is that if a conflict is detected
+ * and the resolution strategy is last_update_wins, then start writing
+ * all the changes to a file similar to what we do for streaming mode.
+ * Once commit_prepared arrives, we will read and apply the changes.
+ */
+ if (twophase && pg_strcasecmp(resolvers[type].resolver, "last_update_wins") == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("Setting any resolver to last_update_wins and %s are mutually exclusive options",
+ "two_phase = true")));
}
/* Once validation is complete, warn users if prerequisites are not met. */
@@ -646,7 +662,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/* Parse and check conflict resolvers. */
- parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers,
+ opts.twophase);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1385,6 +1402,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is
+ * implemented.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable two_phase when a time based resolver is configured")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1640,8 +1670,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
List *conflict_resolvers = NIL;
- /* Get the list of conflict types and resolvers and validate them. */
- conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
+ /*
+ * Get the list of conflict types and resolvers and validate
+ * them.
+ */
+ conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers,
+ sub);
/* Warn users if prerequisites are not met */
conf_detection_check_prerequisites();
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index f944df761c..5b031a462a 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -565,10 +565,20 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
static bool
has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictslot, ConflictType type,
- ConflictResolver resolver, TupleTableSlot *slot,
- Oid subid, bool apply_remote, ItemPointer tupleid)
+ TupleTableSlot *slot, Oid subid, ItemPointer tupleid)
{
+ ConflictResolver resolver;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+
+ /*
+ * Proceed only if the resolver is not set to 'ERROR'; if the resolver is
+ * 'ERROR', the caller will handle it.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type, NULL, NULL,
+ subid);
+ if (resolver == CR_ERROR)
+ return false;
/* Check all the unique indexes for a conflict */
foreach_oid(uniqueidx, conflictindexes)
@@ -584,8 +594,17 @@ has_conflicting_tuple(EState *estate, ResultRelInfo *resultRelInfo,
RepOriginId origin;
TimestampTz committs;
TransactionId xmin;
+ bool apply_remote = false;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(*conflictslot, rel, type,
+ &apply_remote, NULL, subid);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
@@ -634,8 +653,6 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
@@ -650,23 +667,10 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_INSERT_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'insert_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_INSERT_EXISTS, resolver, slot, subid,
- apply_remote, &invalidItemPtr))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, slot, subid,
+ &invalidItemPtr))
return;
/* OK, store the tuple and create index entries for it */
@@ -750,8 +754,6 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
TU_UpdateIndexes update_indexes;
List *conflictindexes;
bool conflict = false;
- ConflictResolver resolver;
- bool apply_remote = false;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -765,23 +767,9 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- /*
- * Get the configured resolver and determine if remote changes should
- * be applied.
- */
- resolver = GetConflictResolver(rel, CT_UPDATE_EXISTS, &apply_remote,
- NULL, subid);
-
- /*
- * Check for conflict and return to caller for resolution if found.
- * Only perform this check if the 'update_exists' resolver is not set
- * to 'ERROR'; if the resolver is 'ERROR', the upcoming
- * CheckAndReportConflict() call will handle it.
- */
- if (resolver != CR_ERROR &&
- has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
- CT_UPDATE_EXISTS, resolver, slot, subid,
- apply_remote, tid))
+ /* Check for conflict and return to caller for resolution if found */
+ if (has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_UPDATE_EXISTS, slot, subid, tid))
return;
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..6cbfab0097 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,18 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Do not start a new parallel worker if 'last_update_wins' is configured
+ * for any conflict type, as we need the commit timestamp in the
+ * beginning.
+ *
+ * XXX: To lift this restriction, we could write the changes to a file
+ * when a conflict is detected, and then at the commit time, let the
+ * remaining changes be applied by the apply worker.
+ */
+ if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 82a3bc26f4..6c5c423fc0 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -44,6 +44,7 @@ static const char *const ConflictTypeNames[] = {
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -69,12 +70,12 @@ static const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, CR_ERROR}
};
/*
@@ -384,6 +385,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -665,6 +671,15 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
+
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -709,6 +724,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -737,7 +788,7 @@ can_create_full_tuple(Relation localrel,
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
List *
-GetAndValidateSubsConflictResolverList(List *stmtresolvers)
+GetAndValidateSubsConflictResolverList(List *stmtresolvers, Subscription *sub)
{
ConflictTypeResolver *conftyperesolver = NULL;
List *res = NIL;
@@ -770,6 +821,21 @@ GetAndValidateSubsConflictResolverList(List *stmtresolvers)
conftyperesolver->conflict_type,
conftyperesolver->resolver);
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * parse_subscription_conflict_resolvers() comments is implemented.
+ */
+ if ((pg_strcasecmp(conftyperesolver->resolver, "last_update_wins") == 0) &&
+ sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver for a subscription that has two_phase enabled",
+ "last_update_wins")));
+
/* Add the conflict type to the list of seen types */
conflictTypes = lappend(conflictTypes,
makeString((char *) conftyperesolver->conflict_type));
@@ -1009,15 +1075,31 @@ RemoveSubscriptionConflictResolvers(Oid subid)
* false otherwise.
*/
ConflictResolver
-GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+GetConflictResolver(TupleTableSlot *localslot, Relation localrel,
+ ConflictType type, bool *apply_remote,
LogicalRepTupleData *newtup, Oid subid)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(localslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1045,3 +1127,40 @@ GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index baf696d8e6..3094030103 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -159,6 +159,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index bfb9c37071..ae75c95131 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1006,6 +1006,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -2738,7 +2744,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
{
TupleTableSlot *newslot;
- resolver = GetConflictResolver(localrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2811,7 +2818,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* the configured resolver is in favor of applying the change, convert
* UPDATE to INSERT and apply the change.
*/
- resolver = GetConflictResolver(localrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -2964,7 +2971,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
- resolver = GetConflictResolver(localrel, CT_DELETE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, localrel,
+ CT_DELETE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -2993,7 +3001,7 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* The tuple to be deleted could not be found. Based on resolver
* configured, either skip and log a message or emit an error.
*/
- resolver = GetConflictResolver(localrel, CT_DELETE_MISSING,
+ resolver = GetConflictResolver(localslot, localrel, CT_DELETE_MISSING,
&apply_remote, NULL,
MySubscription->oid);
@@ -3191,7 +3199,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_MISSING,
+ resolver = GetConflictResolver(localslot, partrel, CT_UPDATE_MISSING,
&apply_remote, newtup,
MySubscription->oid);
@@ -3234,7 +3242,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
newslot = table_slot_create(partrel, &estate->es_tupleTable);
slot_store_data(newslot, part_entry, newtup);
- resolver = GetConflictResolver(partrel, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver = GetConflictResolver(localslot, partrel,
+ CT_UPDATE_ORIGIN_DIFFERS,
&apply_remote, NULL,
MySubscription->oid);
@@ -4766,6 +4775,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4806,10 +4816,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c68bf6fe7e..a62e439af9 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "replication/logicalproto.h"
#include "replication/logicalrelation.h"
@@ -70,6 +71,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,
@@ -110,17 +114,20 @@ extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver *resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictResolvers(Oid confid);
-extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers,
+ Subscription *sub);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
extern void SetDefaultResolvers(ConflictTypeResolver *conflictResolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
-extern ConflictResolver GetConflictResolver(Relation localrel,
+extern ConflictResolver GetConflictResolver(TupleTableSlot *localslot,
+ Relation localrel,
ConflictType type,
bool *apply_remote,
LogicalRepTupleData *newtup,
Oid subid);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
#endif
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 9a9c75b835..7eacc0695c 100755
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -152,6 +152,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -252,6 +280,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -305,16 +363,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -401,10 +491,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -431,6 +525,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
--
2.34.1
v14-0005-Implements-Clock-skew-management-between-nodes.patchapplication/octet-stream; name=v14-0005-Implements-Clock-skew-management-between-nodes.patchDownload
From 050edf4c5166320d79267db6a85c0cf654462095 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 19 Sep 2024 18:14:05 +0530
Subject: [PATCH v14 5/5] Implements Clock-skew management between nodes.
This patch attempts to manage clock skew between nodes by
introducing two new GUCs:
a) max_logical_rep_clock_skew
b) max_logical_rep_clock_skew_action
c) max_logical_rep_clock_skew_wait
If the timestamp of the currently replayed transaction is in the future
compared to the current time on the subscriber and the difference is
larger than 'max_logical_rep_clock_skew', then the action configured
in 'max_logical_rep_clock_skew_action' is performed by the apply worker.
If user configures 'wait' in 'max_logical_rep_clock_skew_action' and
actual clock skew is large while 'max_logical_rep_clock_skew' is small,
the apply worker may have to wait for a longer period to manage the clock
skew. To control this maximum wait time, a new GUC,
'max_logical_rep_clock_skew_wait', is provided. This allows the user to
set a cap on how long the apply worker should wait. If the computed wait
time exceeds this value, the apply worker will error out without waiting.
---
.../replication/logical/applyparallelworker.c | 22 ++-
src/backend/replication/logical/worker.c | 125 +++++++++++++++++-
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_tables.c | 40 ++++++
src/backend/utils/misc/postgresql.conf.sample | 9 +-
src/include/replication/logicalworker.h | 18 +++
src/include/replication/worker_internal.h | 2 +-
src/include/utils/guc.h | 1 +
src/include/utils/timestamp.h | 1 +
src/tools/pgindent/typedefs.list | 1 +
10 files changed, 210 insertions(+), 10 deletions(-)
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index 6cbfab0097..809dab334e 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -314,15 +314,16 @@ pa_can_start(void)
return false;
/*
- * Do not start a new parallel worker if 'last_update_wins' is configured
- * for any conflict type, as we need the commit timestamp in the
- * beginning.
+ * Do not start a new parallel worker if either max clock skew or
+ * 'last_update_wins' is configured for any conflict type. In both of the
+ * cases we need the commit timestamp in the beginning.
*
* XXX: To lift this restriction, we could write the changes to a file
* when a conflict is detected, and then at the commit time, let the
* remaining changes be applied by the apply worker.
*/
- if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ if ((max_logical_rep_clock_skew > LR_CLOCK_SKEW_DEFAULT) ||
+ CheckIfSubHasTimeStampResolver(MySubscription->oid))
return false;
return true;
@@ -709,9 +710,20 @@ pa_process_spooled_messages_if_required(void)
}
else if (fileset_state == FS_READY)
{
+ /*
+ * Currently we do not support starting parallel apply worker when
+ * either clock skew is configured or conflict resolution is
+ * configured to last_update_wins, thus it is okay to pass 0 as
+ * origin-timestamp here.
+ *
+ * XXX: If in future, we support starting pa worker even with
+ * last_update_wins resolver, then here we need to pass remote's
+ * commit/prepare/abort timestamp; we can get that info from leader
+ * worker in shared memory.
+ */
apply_spooled_messages(&MyParallelShared->fileset,
MyParallelShared->xid,
- InvalidXLogRecPtr);
+ InvalidXLogRecPtr, 0);
pa_set_fileset_state(MyParallelShared, FS_EMPTY);
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ae75c95131..de41640fa9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -318,6 +318,20 @@ static uint32 parallel_stream_nchanges = 0;
/* Are we initializing an apply worker? */
bool InitializingApplyWorker = false;
+/*
+ * GUC support
+ */
+const struct config_enum_entry logical_rep_clock_skew_action_options[] = {
+ {"error", LR_CLOCK_SKEW_ACTION_ERROR, false},
+ {"wait", LR_CLOCK_SKEW_ACTION_WAIT, false},
+ {NULL, 0, false}
+};
+
+/* GUCs */
+int max_logical_rep_clock_skew = LR_CLOCK_SKEW_DEFAULT;
+int max_logical_rep_clock_skew_action = LR_CLOCK_SKEW_ACTION_ERROR;
+int max_logical_rep_clock_skew_wait = 300; /* 5 mins */
+
/*
* We enable skipping all data modification changes (INSERT, UPDATE, etc.) for
* the subscription if the remote transaction's finish LSN matches the subskiplsn.
@@ -985,6 +999,95 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
ExecStoreVirtualTuple(slot);
}
+/*
+ * Manage clock skew between nodes.
+ *
+ * It checks if the remote timestamp is ahead of the local clock
+ * and if the difference exceeds max_logical_rep_clock_skew, it performs
+ * the action specified by the max_logical_rep_clock_skew_action.
+ */
+static void
+manage_clock_skew(TimestampTz origin_timestamp)
+{
+ TimestampTz current;
+ TimestampTz delayUntil;
+ long msecs;
+ int rc;
+
+ /* nothing to do if no max clock skew configured */
+ if (max_logical_rep_clock_skew == LR_CLOCK_SKEW_DEFAULT)
+ return;
+
+ current = GetCurrentTimestamp();
+
+ /*
+ * If the timestamp of the currently replayed transaction is in the future
+ * compared to the current time on the subscriber and the difference is
+ * larger than max_logical_rep_clock_skew, then perform the action
+ * specified by the max_logical_rep_clock_skew_action setting.
+ */
+ if (origin_timestamp > current &&
+ TimestampDifferenceExceeds(current, origin_timestamp,
+ max_logical_rep_clock_skew * 1000))
+ {
+ if (max_logical_rep_clock_skew_action == LR_CLOCK_SKEW_ACTION_ERROR)
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew exceeds max_logical_rep_clock_skew (%d seconds)",
+ max_logical_rep_clock_skew)));
+
+ /* Perform the wait */
+ while (true)
+ {
+ delayUntil =
+ TimestampTzMinusSeconds(origin_timestamp,
+ max_logical_rep_clock_skew);
+
+ /* Exit without waiting if it's already past 'delayUntil' time */
+ msecs = TimestampDifferenceMilliseconds(GetCurrentTimestamp(),
+ delayUntil);
+ if (msecs <= 0)
+ break;
+
+ /* The wait time should not exceed max_logical_rep_clock_skew_wait */
+ if (msecs > (max_logical_rep_clock_skew_wait * 1000L))
+ ereport(ERROR,
+ (errcode(ERRCODE_CONFIGURATION_LIMIT_EXCEEDED),
+ errmsg_internal("clock skew wait time exceeds max_logical_rep_clock_skew_wait (%d seconds)",
+ max_logical_rep_clock_skew_wait)));
+
+ elog(DEBUG2, "delaying apply for %ld milliseconds to manage clock skew",
+ msecs);
+
+ /* Sleep until we are signaled or msecs have elapsed */
+ rc = WaitLatch(MyLatch,
+ WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
+ msecs,
+ WAIT_EVENT_LOGICAL_CLOCK_SKEW);
+
+ /* Exit the loop if msecs have elapsed */
+ if (rc & WL_TIMEOUT)
+ break;
+
+ if (rc & WL_LATCH_SET)
+ {
+ ResetLatch(MyLatch);
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ /*
+ * This might change max_logical_rep_clock_skew and
+ * max_logical_rep_clock_skew_wait.
+ */
+ if (ConfigReloadPending)
+ {
+ ConfigReloadPending = false;
+ ProcessConfigFile(PGC_SIGHUP);
+ }
+ }
+ }
+}
+
/*
* Handle BEGIN message.
*/
@@ -1007,6 +1110,9 @@ apply_handle_begin(StringInfo s)
pgstat_report_activity(STATE_RUNNING, NULL);
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.committime);
+
/*
* Capture the commit timestamp of the remote transaction for time based
* conflict resolution purpose.
@@ -1069,6 +1175,9 @@ apply_handle_begin_prepare(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /* Check if there is any clock skew and perform configured action */
+ manage_clock_skew(begin_data.prepare_time);
}
/*
@@ -1324,7 +1433,8 @@ apply_handle_stream_prepare(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset,
- prepare_data.xid, prepare_data.prepare_lsn);
+ prepare_data.xid, prepare_data.prepare_lsn,
+ prepare_data.prepare_time);
/* Mark the transaction as prepared. */
apply_handle_prepare_internal(&prepare_data);
@@ -2029,7 +2139,8 @@ ensure_last_message(FileSet *stream_fileset, TransactionId xid, int fileno,
*/
void
apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn)
+ XLogRecPtr lsn,
+ TimestampTz origin_timestamp)
{
int nchanges;
char path[MAXPGPATH];
@@ -2082,6 +2193,13 @@ apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
end_replication_step();
+ /*
+ * If origin_timestamp is provided by caller, then check clock skew with
+ * respect to the passed time and take configured action.
+ */
+ if (origin_timestamp)
+ manage_clock_skew(origin_timestamp);
+
/*
* Read the entries one by one and pass them through the same logic as in
* apply_dispatch.
@@ -2187,7 +2305,8 @@ apply_handle_stream_commit(StringInfo s)
* spooled operations.
*/
apply_spooled_messages(MyLogicalRepWorker->stream_fileset, xid,
- commit_data.commit_lsn);
+ commit_data.commit_lsn,
+ commit_data.committime);
apply_handle_commit_internal(&commit_data);
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 8efb4044d6..0ebad6fcab 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -59,6 +59,7 @@ CHECKPOINTER_MAIN "Waiting in main loop of checkpointer process."
LOGICAL_APPLY_MAIN "Waiting in main loop of logical replication apply process."
LOGICAL_LAUNCHER_MAIN "Waiting in main loop of logical replication launcher process."
LOGICAL_PARALLEL_APPLY_MAIN "Waiting in main loop of logical replication parallel apply process."
+LOGICAL_CLOCK_SKEW "Waiting in apply-begin of logical replication apply process to bring clock skew in permissible range."
RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive, during streaming recovery."
REPLICATION_SLOTSYNC_MAIN "Waiting in main loop of slot sync worker."
REPLICATION_SLOTSYNC_SHUTDOWN "Waiting for slot sync worker to shut down."
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309db58..c768a11963 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -68,6 +68,7 @@
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
+#include "replication/logicalworker.h"
#include "replication/slot.h"
#include "replication/slotsync.h"
#include "replication/syncrep.h"
@@ -482,6 +483,7 @@ extern const struct config_enum_entry archive_mode_options[];
extern const struct config_enum_entry recovery_target_action_options[];
extern const struct config_enum_entry wal_sync_method_options[];
extern const struct config_enum_entry dynamic_shared_memory_options[];
+extern const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* GUC option variables that are exported from this module
@@ -3714,6 +3716,33 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets maximum clock skew tolerance between logical "
+ "replication nodes beyond which action configured "
+ "in max_logical_rep_clock_skew_action is triggered."),
+ gettext_noop("-1 turns this check off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew,
+ LR_CLOCK_SKEW_DEFAULT, LR_CLOCK_SKEW_DEFAULT, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"max_logical_rep_clock_skew_wait", PGC_SIGHUP, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets max limit on how long apply worker shall wait to "
+ "bring clock skew within permissible range of max_logical_rep_clock_skew. "
+ "If the computed wait time is more than this value, "
+ "apply worker will error out without waiting."),
+ gettext_noop("0 turns this limit off."),
+ GUC_UNIT_S
+ },
+ &max_logical_rep_clock_skew_wait,
+ 300, 0, 3600,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, 0, 0, NULL, NULL, NULL
@@ -4991,6 +5020,17 @@ struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"max_logical_rep_clock_skew_action", PGC_POSTMASTER, REPLICATION_SUBSCRIBERS,
+ gettext_noop("Sets the action to perform if a clock skew higher "
+ "than max_logical_rep_clock_skew is detected."),
+ NULL
+ },
+ &max_logical_rep_clock_skew_action,
+ LR_CLOCK_SKEW_ACTION_ERROR, logical_rep_clock_skew_action_options,
+ NULL, NULL, NULL
+ },
+
{
{"track_functions", PGC_SUSET, STATS_CUMULATIVE,
gettext_noop("Collects function-level statistics on database activity."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..6424432362 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -383,7 +383,14 @@
# (change requires restart)
#max_sync_workers_per_subscription = 2 # taken from max_logical_replication_workers
#max_parallel_apply_workers_per_subscription = 2 # taken from max_logical_replication_workers
-
+#max_logical_rep_clock_skew = -1 # maximum clock skew tolerance between logical
+ # replication nodes beyond which action configured in
+ # 'max_logical_rep_clock_skew_action' is triggered.
+#max_logical_rep_clock_skew_action = error # error or wait
+ # (change requires restart)
+#max_logical_rep_clock_skew_wait = 300 # max limit on how long apply worker
+ # shall wait to bring clock skew within permissible
+ # range of max_logical_rep_clock_skew.
#------------------------------------------------------------------------------
# QUERY TUNING
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index a18d79d1b2..7cb03062ac 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -14,7 +14,25 @@
#include <signal.h>
+/*
+ * The default for max_logical_rep_clock_skew is -1, which means ignore clock
+ * skew (the check is turned off).
+ */
+#define LR_CLOCK_SKEW_DEFAULT -1
+
+/*
+ * Worker Clock Skew Action.
+ */
+typedef enum
+{
+ LR_CLOCK_SKEW_ACTION_ERROR,
+ LR_CLOCK_SKEW_ACTION_WAIT,
+} LogicalRepClockSkewAction;
+
extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending;
+extern PGDLLIMPORT int max_logical_rep_clock_skew;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_action;
+extern PGDLLIMPORT int max_logical_rep_clock_skew_wait;
extern void ApplyWorkerMain(Datum main_arg);
extern void ParallelApplyWorkerMain(Datum main_arg);
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index 9646261d7e..95b2a5286d 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -268,7 +268,7 @@ extern void stream_stop_internal(TransactionId xid);
/* Common streaming function to apply all the spooled messages */
extern void apply_spooled_messages(FileSet *stream_fileset, TransactionId xid,
- XLogRecPtr lsn);
+ XLogRecPtr lsn, TimestampTz origin_timestamp);
extern void apply_dispatch(StringInfo s);
diff --git a/src/include/utils/guc.h b/src/include/utils/guc.h
index 840b0fe57f..837a6ec713 100644
--- a/src/include/utils/guc.h
+++ b/src/include/utils/guc.h
@@ -317,6 +317,7 @@ extern PGDLLIMPORT const struct config_enum_entry dynamic_shared_memory_options[
extern PGDLLIMPORT const struct config_enum_entry recovery_target_action_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_level_options[];
extern PGDLLIMPORT const struct config_enum_entry wal_sync_method_options[];
+extern PGDLLIMPORT const struct config_enum_entry logical_rep_clock_skew_action_options[];
/*
* Functions exported by guc.c
diff --git a/src/include/utils/timestamp.h b/src/include/utils/timestamp.h
index a6ce03ed46..53b828d89d 100644
--- a/src/include/utils/timestamp.h
+++ b/src/include/utils/timestamp.h
@@ -84,6 +84,7 @@ IntervalPGetDatum(const Interval *X)
/* Macros for doing timestamp arithmetic without assuming timestamp's units */
#define TimestampTzPlusMilliseconds(tz,ms) ((tz) + ((ms) * (int64) 1000))
#define TimestampTzPlusSeconds(tz,s) ((tz) + ((s) * (int64) 1000000))
+#define TimestampTzMinusSeconds(tz,s) ((tz) - ((s) * (int64) 1000000))
/* Set at postmaster start */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index bbf3221075..ca10602af7 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1568,6 +1568,7 @@ LogicalOutputPluginWriterPrepareWrite
LogicalOutputPluginWriterUpdateProgress
LogicalOutputPluginWriterWrite
LogicalRepBeginData
+LogicalRepClockSkewAction
LogicalRepCommitData
LogicalRepCommitPreparedTxnData
LogicalRepCtxStruct
--
2.34.1
On Thu, Sep 19, 2024 at 5:43 PM vignesh C <vignesh21@gmail.com> wrote:
I was reviewing the CONFLICT RESOLVER (insert_exists='apply_remote')
and found that one conflict remains unresolved in the following
scenario:
Thanks for the review and testing.
Pub:
CREATE TABLE circles(c1 CIRCLE, c2 text, EXCLUDE USING gist (c1 WITH &&));
CREATE PUBLICATION pub1 for table circles;Sub:
CREATE TABLE circles(c1 CIRCLE, c2 text, EXCLUDE USING gist (c1 WITH &&))
insert into circles values('<(0,0), 5>', 'sub');
CREATE SUBSCRIPTION ... PUBLICATION pub1 CONFLICT RESOLVER
(insert_exists='apply_remote');The following conflict is not detected and resolved with remote tuple data:
Pub:
INSERT INTO circles VALUES('<(0,0), 5>', 'pub');2024-09-19 17:32:36.637 IST [31463] 31463 LOG: conflict detected on
relation "public.t1": conflict=insert_exists, Resolution=apply_remote.
2024-09-19 17:32:36.637 IST [31463] 31463 DETAIL: Key already
exists in unique index "t1_pkey", modified in transaction 742,
applying the remote changes.
Key (c1)=(1); existing local tuple (1, sub); remote tuple (1, pub).
2024-09-19 17:32:36.637 IST [31463] 31463 CONTEXT: processing
remote data for replication origin "pg_16398" during message type
"INSERT" for replication target relation "public.t1" in transaction
744, finished at 0/1528E88
........
2024-09-19 17:32:44.653 IST [31463] 31463 ERROR: conflicting key
value violates exclusion constraint "circles_c1_excl"
2024-09-19 17:32:44.653 IST [31463] 31463 DETAIL: Key
(c1)=(<(0,0),5>) conflicts with existing key (c1)=(<(0,0),5>).
........
We don't support conflict detection for exclusion constraints yet.
Please see the similar issue raised in the conflict-detection thread
and the responses at [1]/messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com and [2]/messages/by-id/CAA4eK1KwqAUGDV3trUZf4hkrUYO3yzwjmBqYtoyFAPMFXpHy3g@mail.gmail.com. Also see the docs at [3]https://www.postgresql.org/docs/devel/logical-replication-conflicts.html <See this in doc: Note that there are other conflict scenarios, such as exclusion constraint violations. Currently, we do not provide additional details for them in the log.>.
[1]: /messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com
[2]: /messages/by-id/CAA4eK1KwqAUGDV3trUZf4hkrUYO3yzwjmBqYtoyFAPMFXpHy3g@mail.gmail.com
[3]: https://www.postgresql.org/docs/devel/logical-replication-conflicts.html <See this in doc: Note that there are other conflict scenarios, such as exclusion constraint violations. Currently, we do not provide additional details for them in the log.>
<See this in doc: Note that there are other conflict scenarios, such
as exclusion constraint violations. Currently, we do not provide
additional details for them in the log.>
thanks
Shveta
On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
On Wed, Sep 18, 2024 at 10:46 AM vignesh C <vignesh21@gmail.com> wrote:
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type + { + AlterSubscriptionStmt *n = + makeNode(AlterSubscriptionStmt); + + n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER; + n->subname = $3; + n->conflict_type = $8; + $$ = (Node *) n; + } + ; +conflict_type: + Sconst { $$ = $1; } + | NULL_P { $$ = NULL; } ;May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;Fixed.
Few comments: 1) This should be in (fout->remoteVersion >= 180000) check to support dumping backward compatible server objects, else dump with older version will fail: + /* Populate conflict type fields using the new query */ + confQuery = createPQExpBuffer(); + appendPQExpBuffer(confQuery, + "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict " + "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid); + confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK); + + ntuples = PQntuples(confRes); + for (j = 0; j < ntuples; j++)2) Can we check and throw an error before the warning is logged in
this case as it seems strange to throw a warning first and then an
error for the same track_commit_timestamp configuration:
postgres=# create subscription sub1 connection ... publication pub1
conflict resolver (insert_exists = 'last_update_wins');
WARNING: conflict detection and resolution could be incomplete due to
disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs
cannot be detected, and the origin and commit timestamp for the local
row will not be logged.
ERROR: resolver last_update_wins requires "track_commit_timestamp" to
be enabled
HINT: Make sure the configuration parameter "track_commit_timestamp" is set.Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].
Clarification:
The fixes for mentioned comments from Vignesh - [1] & [2] are fixed in
patch-001. Thank you Ajin for providing the changes.
Show quoted text
New in patches:
1) Added partition table tests in 034_conflict_resolver.pl in 002 and
003 patches.
2) 003 has a bug fix for update_exists conflict resolution on
partitioned tables.[1]: /messages/by-id/CALDaNm3es1JqU8Qcv5Yw=7Ts2dOvaV8a_boxPSdofB+DTx1oFg@mail.gmail.com
[2]: /messages/by-id/CALDaNm18HuAcNsEC47J6qLRC7rMD2Q9_wT_hFtcc4UWqsfkgjA@mail.gmail.comThanks,
Nisha
On Fri, Sep 13, 2024 at 10:20 PM vignesh C <vignesh21@gmail.com> wrote:
Few comments:
1) Tab completion missing for:
a) ALTER SUBSCRIPTION name CONFLICT RESOLVER
b) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
c) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR
Added.
2) Documentation missing for:
a) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
b) ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR
Added.
3) This reset is not required here, if valid was false it would have thrown an error and exited: a) + if (!valid) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("%s is not a valid conflict type", conflict_type)); + + /* Reset */ + valid = false;b) Similarly here too: + if (!valid) + ereport(ERROR, + errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("%s is not a valid conflict resolver", conflict_resolver)); + + /* Reset */ + valid = false;
Actually, the reset is for when valid becomes true. I think it it is
required here.
4) How about adding CT_MAX inside the enum itself as the last enum value:
typedef enum
{
/* The row to be inserted violates unique constraint */
CT_INSERT_EXISTS,/* The row to be updated was modified by a different origin */
CT_UPDATE_ORIGIN_DIFFERS,/* The updated row value violates unique constraint */
CT_UPDATE_EXISTS,/* The row to be updated is missing */
CT_UPDATE_MISSING,/* The row to be deleted was modified by a different origin */
CT_DELETE_ORIGIN_DIFFERS,/* The row to be deleted is missing */
CT_DELETE_MISSING,/*
* Other conflicts, such as exclusion constraint violations, involve more
* complex rules than simple equality checks. These conflicts are left for
* future improvements.
*/
} ConflictType;#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
/* Min and max conflict type */
#define CT_MIN CT_INSERT_EXISTS
#define CT_MAX CT_DELETE_MISSINGand the for loop can be changed to:
for (type = 0; type < CT_MAX; type++)This way CT_MIN can be removed and CT_MAX need not be changed every
time a new enum is added.Also the following +1 can be removed from the variables:
ConflictTypeResolver conflictResolvers[CT_MAX + 1];
I tried changing this, but the enums are used in swicth cases and this
throws a compiler warning that CT_MAX is not checked in the switch case.
However, I have changed the use of (CT_MAX +1) and instead used
CONFLICT_NUM_TYPES in those places.
5) Similar thing can be done with ConflictResolver enum too. i.e
remove CR_MIN and add CR_MAX as the last element of enum
typedef enum ConflictResolver
{
/* Apply the remote change */
CR_APPLY_REMOTE = 1,/* Keep the local change */
CR_KEEP_LOCAL,/* Apply the remote change; skip if it can not be applied */
CR_APPLY_OR_SKIP,/* Apply the remote change; emit error if it can not be applied */
CR_APPLY_OR_ERROR,/* Skip applying the change */
CR_SKIP,/* Error out */
CR_ERROR,
} ConflictResolver;/* Min and max conflict resolver */
#define CR_MIN CR_APPLY_REMOTE
#define CR_MAX CR_ERROR
same as previous comment.
6) Except scansup.h inclusion, other inclusions added are not required
in subscriptioncmds.c file.7)The inclusions "access/heaptoast.h", "access/table.h",
"access/tableam.h", "catalog/dependency.h",
"catalog/pg_subscription.h", "catalog/pg_subscription_conflict.h" and
"catalog/pg_inherits.h" are not required in conflict.c file.
Removed.
8) Can we change this to use the new foreach_ptr implementations added: + foreach(lc, stmtresolvers) + { + DefElem *defel = (DefElem *) lfirst(lc); + ConflictType type; + char *resolver;to use foreach_ptr like:
foreach_ptr(DefElem, defel, stmtresolvers)
{
+ ConflictType type;
+ char *resolver;
....
}
Changed accordingly.
regards,
Ajin Cherian
Fujitsu Australia
On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
On Wed, Sep 18, 2024 at 10:46 AM vignesh C <vignesh21@gmail.com> wrote:
On Thu, 12 Sept 2024 at 14:03, Ajin Cherian <itsajin@gmail.com> wrote:
On Tue, Sep 3, 2024 at 7:42 PM vignesh C <vignesh21@gmail.com> wrote:
On Fri, 30 Aug 2024 at 11:01, Nisha Moond <nisha.moond412@gmail.com> wrote:
Here is the v11 patch-set. Changes are:
1) This command crashes:
ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR NULL;
#0 __strcmp_avx2 () at ../sysdeps/x86_64/multiarch/strcmp-avx2.S:116
#1 0x000055c67270600a in ResetConflictResolver (subid=16404,
conflict_type=0x0) at conflict.c:744
#2 0x000055c67247e0c3 in AlterSubscription (pstate=0x55c6748ff9d0,
stmt=0x55c67497dfe0, isTopLevel=true) at subscriptioncmds.c:1664+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type + { + AlterSubscriptionStmt *n = + makeNode(AlterSubscriptionStmt); + + n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER; + n->subname = $3; + n->conflict_type = $8; + $$ = (Node *) n; + } + ; +conflict_type: + Sconst { $$ = $1; } + | NULL_P { $$ = NULL; } ;May be conflict_type should be changed to:
+conflict_type:
+ Sconst
{ $$ = $1; }
;Fixed.
Few comments: 1) This should be in (fout->remoteVersion >= 180000) check to support dumping backward compatible server objects, else dump with older version will fail: + /* Populate conflict type fields using the new query */ + confQuery = createPQExpBuffer(); + appendPQExpBuffer(confQuery, + "SELECT confrtype, confrres FROM pg_catalog.pg_subscription_conflict " + "WHERE confsubid = %u;", subinfo[i].dobj.catId.oid); + confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK); + + ntuples = PQntuples(confRes); + for (j = 0; j < ntuples; j++)2) Can we check and throw an error before the warning is logged in
this case as it seems strange to throw a warning first and then an
error for the same track_commit_timestamp configuration:
postgres=# create subscription sub1 connection ... publication pub1
conflict resolver (insert_exists = 'last_update_wins');
WARNING: conflict detection and resolution could be incomplete due to
disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs
cannot be detected, and the origin and commit timestamp for the local
row will not be logged.
ERROR: resolver last_update_wins requires "track_commit_timestamp" to
be enabled
HINT: Make sure the configuration parameter "track_commit_timestamp" is set.Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].
Just noticed that there are failures for the new
034_conflict_resolver.pl test on CFbot. From the initial review it
seems to be a test issue and not a bug.
We will fix these along with the next version of patch-sets.
Thanks,
Nisha
On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].
Thanks for the patches. I am reviewing patch001, it is WIP, but please
find initial set of comments:
1)
Please see these 2 errors:
postgres=# create subscription sub2 connection '....' publication pub1
CONFLICT RESOLVER(insert_exists = 'error') WITH (two_phase=true,
streaming=ON, streaming=OFF);
ERROR: conflicting or redundant options
LINE 1: ...ists='error') WITH (two_phase=true, streaming=ON, streaming=...
^
postgres=# create subscription sub2 connection '....' publication pub1
CONFLICT RESOLVER(insert_exists = 'error', insert_exists = 'error')
WITH (two_phase=true);
ERROR: duplicate conflict type "insert_exists" found
When we give duplicate options in 'WITH', we get an error as
'conflicting or redundant options' with 'position' pointed out, while
in case of CONFLICT RESOLVER, it is different. Can we review to see if
we can have similar error in CONFLICT RESOLVER as that of WITH?
Perhaps we need to call 'errorConflictingDefElem' from resolver flow.
2)
+static void
+parse_subscription_conflict_resolvers(List *stmtresolvers,
+ ConflictTypeResolver *resolvers)
+{
+ ListCell *lc;
+ List *SeenTypes = NIL;
+
+
Remove redundant blank line
3)
parse_subscription_conflict_resolvers():
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+}
Remove redundant blank line
4)
parse_subscription_conflict_resolvers():
+ resolver = defGetString(defel);
+ type = validate_conflict_type_and_resolver(defel->defname,
+ defGetString(defel));
Shall we use 'resolver' as arg to validate function instead of doing
defGetStringagain?
5)
parse_subscription_conflict_resolvers():
+ /* Update the corresponding resolver for the given conflict type. */
+ resolvers[type].resolver = downcase_truncate_identifier(resolver,
strlen(resolver), false);
Shouldn't we do this before validate_conflict_type_and_resolver()
itself like we do it in GetAndValidateSubsConflictResolverList()? And
do we need downcase_truncate_identifier on defel->defname as well
before we do validate_conflict_type_and_resolver()?
6)
GetAndValidateSubsConflictResolverList() and
parse_subscription_conflict_resolvers() are similar but yet have so
many differences which I pointed out above. Not a good idea to
maintain 2 such functions. We should have a common parsing function
for both Create and Alter Sub. Can you please review the possibility
of that?
~~
conflict.c:
7)
+
+
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
Remove redundant blank line
8)
* Set default values for CONFLICT RESOLVERS for each conflict type
Is it better to change to: Set default resolver for each conflict type
9)
validate_conflict_type_and_resolver(): Since it is called from other
file as well, shall we rename to ValidateConflictTypeAndResolver()
10)
+ return type;
+
+}
Remove redundant blank line after 'return'
thanks
Shveta
On Thu, Sep 26, 2024 at 2:57 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 20, 2024 at 8:40 AM Nisha Moond <nisha.moond412@gmail.com> wrote:
Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].Thanks for the patches. I am reviewing patch001, it is WIP, but please
find initial set of comments:
Please find next set of comments on patch001:
11)
conflict.c
#include "access/tableam.h" (existing)
#include "replication/logicalproto.h" (added by patch002)
Above 2 are not needed. The code compiles without these. I think the
first one has become redundant due to inclusion of other header files
which indirectly include this.
12)
create_subscription.sgml:
+ apply_remote (enum)
+ This resolver applies the remote change. It can be used for
insert_exists, update_exists, update_origin_differs and
delete_origin_differs. It is the default resolver for insert_exists
and update_exists.
Wrong info, it is default for update_origin_differs and delete_origin_differs
13)
alter_subscription.sgml:
Synopsis:
+ ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR (conflict_type)
we don't support parenthesis in the syntax. So please correct the doc.
postgres=# ALTER SUBSCRIPTION sub1 RESET CONFLICT RESOLVER FOR
('insert_exists');
ERROR: syntax error at or near "("
14)
alter_subscription.sgml:
+ CONFLICT RESOLVER ( conflict_type [= conflict_resolver] [, ... ] )
+ This clause alters either the default conflict resolvers or those
set by CREATE SUBSCRIPTION. Refer to section CONFLICT RESOLVERS for
the details on supported conflict_types and conflict_resolvers.
+ conflict_type
+ The conflict type being reset to its default resolver setting. For
details on conflict types and their default resolvers, refer to
section CONFLICT RESOLVERS
a) These details seem problematic. Shouldn't we have RESET as heading
similar to SKIP and then try explaining both ALL and conflict_type
under that. Above seems we are trying to explain conflict_type of
'CONFLICT RESOLVER ( conflict_type [= conflict_resolver]' subcommand
while
giving details of RESET subcommand.
b) OTOH, 'CONFLICT RESOLVER ( conflict_type [= conflict_resolver]'
should have its own explanation of conflict_type and conflict_resolver
parameters.
15)
logical-replication.sgml:
Existing:
+ Additional logging is triggered in various conflict scenarios, each
identified as a conflict type, and the conflict statistics are
collected (displayed in the pg_stat_subscription_stats view). Users
have the option to configure a conflict_resolver for each
conflict_type when creating a subscription. For more information on
the supported conflict_types detected and conflict_resolvers, refer to
section CONFLICT RESOLVERS.
Suggestion:
Additional logging is triggered for various conflict scenarios, each
categorized by a specific conflict type, with conflict statistics
being gathered and displayed in the pg_stat_subscription_stats view.
Users can configure a conflict_resolver for each conflict_type when
creating a subscription.
For more details on the supported conflict types and corresponding
conflict resolvers, refer to the section on <CONFLICT RESOLVERS>.
thanks
Shveta
Here are some review comments for v14-0001.
This is a WIP, but here are my comments for all the SGML parts.
(There will be some overlap here with comments already posted by Shveta)
======
1. file modes after applying the patch
mode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgml
What's going on here? Why are those SGMLs changed to executable?
======
Commit message
2.
nit - a missing period in the first sentence
nit - typo /reseting/resetting/
======
doc/src/sgml/logical-replication.sgml
3.
- <title>Conflicts</title>
+ <title>Conflicts and conflict resolution</title>
nit - change the capitalisation to "and Conflict Resolution" to match
other titles.
~~~
4.
+ Additional logging is triggered in various conflict scenarios,
each identified as a
+ conflict type, and the conflict statistics are collected (displayed in the
+ <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link>
view).
+ Users have the option to configure a
<literal>conflict_resolver</literal> for each
+ <literal>conflict_type</literal> when creating a subscription.
+ For more information on the supported
<literal>conflict_types</literal> detected and
+ <literal>conflict_resolvers</literal>, refer to section
+ <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT
RESOLVERS</literal></link>.
+
nit - "Additional logging is triggered" sounds strange. I reworded
this in the nits attachment. Please see if you approve.
nit - The "conflict_type" and "conflict_resolver" you are referring to
here are syntax elements of the CREATE SUBSCRIPTION, so here I think
they should just be called (without the underscores) "conflict type"
and "conflict resolver".
nit - IMO this would be better split into multiple paragraphs.
nit - There is no such section called "CONFLICT RESOLVERS". I reworded
this link text.
======
doc/src/sgml/monitoring.sgml
5.
The changes here all render with the link including the type "(enum)"
displayed, which I thought it unnecessary/strange.
For example:
See insert_exists (enum) for details about this conflict.
IIUC there is no problem here, but maybe the other end of the link
needed to define xreflabels. I have made the necessary modifications
in the create_subscription.sgml.
======
doc/src/sgml/ref/alter_subscription.sgml
6.
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable>
CONFLICT RESOLVER ( <replaceable
class="parameter">conflict_type</replaceable> [= <replaceable
class="parameter">conflict_resolver</replaceable>] [, ...] )
This syntax seems wrong to me.
Currently, it says:
ALTER SUBSCRIPTION name CONFLICT RESOLVER ( conflict_type [=
conflict_resolver] [, ...] )
But, shouldn't that say:
ALTER SUBSCRIPTION name CONFLICT RESOLVER ( conflict_type =
conflict_resolver [, ...] )
~~~
7.
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable>
RESET CONFLICT RESOLVER FOR (<replaceable
class="parameter">conflict_type</replaceable>)
I can see that this matches the implementation, but I was wondering
why don't you permit resetting multiple conflict_types at the same
time. e.g. what if I want to reset some but not ALL?
~~~
nit - there are some minor whitespace indent problems in the SGML
~~~
8.
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable
class="parameter">conflict_type</replaceable> [= <replaceable
class="parameter">conflict_resolver</replaceable>] [, ... ]
)</literal></term>
+ <listitem>
+ <para>
+ This clause alters either the default conflict resolvers or
those set by <xref linkend="sql-createsubscription"/>.
+ Refer to section <link
linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT
RESOLVERS</literal></link>
+ for the details on supported <literal>conflict_types</literal>
and <literal>conflict_resolvers</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type being reset to its default resolver setting.
+ For details on conflict types and their default resolvers, refer
to section <link
linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT
RESOLVERS</literal></link>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
This section seems problematic:
e.g the syntax seems wrong same as before.
~
There are other nits.
(I've given a rough fix in the nits attachment. Please see it and make
it better).
nit - why do you care if it is "either the default conflict resolvers
or those set...". Why not just say "current resolver"
nit - it does not mention 'conflict_resolver' type in the normal way
nit - there is no actual section called "CONFLICT RESOLVERS"
nit - the part that says "The conflict type being reset to its default
resolver setting." is bogus for this form of the ALTER statement.
~~~
9.
There is no description for the "RESET CONFLICT RESOLVER ALL"
~~~
10.
There is no description for the "RESET CONFLICT RESOLVER FOR (conflict_type)"
======
doc/src/sgml/ref/create_subscription.sgml
11. General - Order
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable
class="parameter">conflict_type</replaceable> = <replaceable
nit - IMO this entire new entry about "CONFLICT RESOLVER" should
appear on the page *above* the "WITH" section, because that is the
order that it is defined in the CREATE SUBSCRIPTION syntax.
~~~
12. General - whitespace
nit - Much of this new section seems to have a slightly wrong
indentation in the SGML. Mostly it is out by 1 or 2 spaces.
~~~
13. General - ordering of conflict_type.
nit - Instead of just some apparent random order, let's put each
insert/update/delete conflict type in alphabetical order, so at least
users can find them where they would expect to find them.
~~~
14.
99. General - ordering of conflict_resolver
nit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.
~~~
15.
+ <para>
+ This optional clause specifies options for conflict resolvers
for different conflict_types.
+ </para>
nit - IMO we don't need the words "options for" here.
~~~
16.
+ <para>
+ The <replaceable class="parameter">conflict_type</replaceable>
and their default behaviour are listed below.
nit - sounded strange to me. reworded it slightly.
~~~
17.
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-insert-exists">
nit - Here, and for all other conflict types, add "xreflabel". See my
review comment #5 for the reason why.
~~~
18.
+ <para>
+ The <replaceable
class="parameter">conflict_resolver</replaceable> and their behaviour
+ are listed below. Users can use any of the following resolvers
for automatic conflict
+ resolution.
+ <variablelist>
nit - reworded this too, to be like the previous review comment.
~~~
19. General - readability.
19a.
IMO the information about what are the default resolvers for each
conflict type, and what resolvers are allowed for each conflict type
should ideally be documented in a tabular form.
Maybe all information is already present in the current document, but
it is certainly hard to easily see it.
As an example, I have added a table in this section. Maybe it is the
best placement for this table, but I gave it mostly how you can
present the same information so it is easier to read.
~
19b.
Bug. In doing this exercise I discovered there are 2 resolvers
("error" and "apply_remote") that both claim to be defaults for the
same conflict types.
They both say:
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
Anyway, this demonstrates that the current information was hard to read.
I can tell from the code implementation what the document was supposed
to say, but I will leave it to the patch authors to fix this one.
(e.g. "apply_remote" says the wrong defaults)
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
PS_NITPICKS_CDR_v140001_DOCS.txttext/plain; charset=US-ASCII; name=PS_NITPICKS_CDR_v140001_DOCS.txtDownload
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index f82adfc..2077c6a 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts and conflict resolution</title>
+ <title>Conflicts and Conflict Resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,18 +1582,19 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered in various conflict scenarios, each identified as a
- conflict type, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view).
- Users have the option to configure a <literal>conflict_resolver</literal> for each
- <literal>conflict_type</literal> when creating a subscription.
- For more information on the supported <literal>conflict_types</literal> detected and
- <literal>conflict_resolvers</literal>, refer to section
- <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>.
-
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ There are various conflict scenarios, each identified as a <firstterm>conflict type</firstterm>.
+ Users can configure a <firstterm>conflict resolver</firstterm> for each
+ conflict type when creating a subscription. For more information, refer to
+ <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIPTION ... CONFLICT RESOLVER</command></link>.
+ </para>
+ <para>
+ When a conflict occurs the details about it are logged, and the conflict
+ statistics are recorded in the <link linkend="monitoring-pg-stat-subscription-stats">
+ <structname>pg_stat_subscription_stats</structname></link> view.
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the
+ log.
</para>
<para>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 55eae8b..1982130 100755
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -353,22 +353,32 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFL
<term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term>
<listitem>
<para>
- This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>.
- Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
- for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>.
+ This clause alters the current conflict resolver for the specified conflict types.
+ Refer to <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIBER ... CONFLICT RESOLVER</command></link>
+ for details about different <literal>conflict_type</literal> and what
+ kind of <literal>conflict_resolver</literal> can be assigned to them.
</para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-resolver">
+ <term><replaceable class="parameter">conflict_resolver</replaceable></term>
+ <listitem>
+ <para>
+ The conflict resolver to use for this conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
</listitem>
</varlistentry>
-
- <varlistentry id="sql-altersubscription-params-conflict-type">
- <term><replaceable class="parameter">conflict_type</replaceable></term>
- <listitem>
- <para>
- The conflict type being reset to its default resolver setting.
- For details on conflict types and their default resolvers, refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>
- </para>
- </listitem>
- </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 25d4c0b..e3d435a 100755
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -98,6 +98,202 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The default behavior for each <replaceable class="parameter">conflict_type</replaceable> is listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists" xreflabel="insert_exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists" xreflabel="update_exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing" xreflabel="update_missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs" xreflabel="update_origin_differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing" xreflabel="delete_missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs" xreflabel="delete_origin_differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ The behavior of each <replaceable class="parameter">conflict_resolver</replaceable>
+ is described below. Users can choose from the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <table id="sql-createsubscription-params-conflict-type-resolver-summary">
+ <title>Conflict type/resolver Summary</title>
+ <tgroup cols="3">
+ <thead>
+ <row><entry>Conflict type</entry> <entry>Default resolver</entry> <entry>Possible resolvers</entry></row>
+ </thead>
+ <tbody>
+ <row><entry>insert_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_missing</entry> <entry>skip</entry> <entry>apply_or_error, apply_or_skip, error, skip</entry></row>
+ <row><entry>update_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>delete_missing</entry> <entry>skip</entry> <entry>error, skip</entry></row>
+ <row><entry>delete_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createsubscription-params-with">
<term><literal>WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
<listitem>
@@ -433,183 +629,6 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
- <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
- <listitem>
- <para>
- This optional clause specifies options for conflict resolvers for different conflict_types.
- </para>
-
- <para>
- The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.
- <variablelist>
- <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists">
- <term><literal>insert_exists</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when inserting a row that violates a
- <literal>NOT DEFERRABLE</literal> unique constraint.
- To log the origin and commit timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually or the resolver is configured to a
- non-default value that can automatically resolve the conflict.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs">
- <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when updating a row that was previously
- modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists">
- <term><literal>update_exists</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when the updated value of a row violates
- a <literal>NOT DEFERRABLE</literal>
- unique constraint. To log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually or the resolver is configured to a
- non-default value that can automatically resolve the conflict.
- Note that when updating a partitioned table, if the updated row
- value satisfies another partition constraint resulting in the
- row being inserted into a new partition, the <literal>insert_exists</literal>
- conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing">
- <term><literal>update_missing</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when the tuple to be updated was not found.
- The update will simply be skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs">
- <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when deleting a row that was previously modified
- by another origin. Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing">
- <term><literal>delete_missing</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This conflict occurs when the tuple to be deleted was not found.
- The delete will simply be skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
-
- </variablelist></para>
-
- <para>
- The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour
- are listed below. Users can use any of the following resolvers for automatic conflict
- resolution.
- <variablelist>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
- <term><literal>error</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This resolver throws an error and stops replication. It can be used for
- any conflict type.
- It is the default resolver for <literal>insert_exists</literal> and
- <literal>update_exists</literal>.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
- <term><literal>skip</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This resolver skips processing the remote change and continue replication
- with the next change.
- It can be used for <literal>update_missing</literal> and
- <literal>delete_missing</literal> and is the default resolver for these types.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
- <term><literal>apply_remote</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This resolver applies the remote change. It can be used for
- <literal>insert_exists</literal>, <literal>update_exists</literal>,
- <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
-
- It is the default resolver for <literal>insert_exists</literal> and
- <literal>update_exists</literal>.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
- <term><literal>keep_local</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- With this resolver, the remote change is not applied and the local tuple is maintained.
- It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
- <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
- <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This resolver is only used for <literal>update_missing</literal>.
- An attempt is made to convert the update into an insert; if this
- cannot be done due to missing information, then the change is skipped.
- </para>
- </listitem>
- </varlistentry>
-
- <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
- <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
- <listitem>
- <para>
- This resolver is only used for <literal>update_missing</literal>.
- An attempt is made to convert the update into an insert; if this cannot
- be done due to missing information, then an error is thrown, and replication
- is stopped.
- </para>
- </listitem>
- </varlistentry>
- </variablelist></para>
-
- </listitem>
- </varlistentry>
</variablelist>
<para>
On Fri, Sep 27, 2024 at 10:44 AM shveta malik <shveta.malik@gmail.com> wrote:
Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].Thanks for the patches. I am reviewing patch001, it is WIP, but please
find initial set of comments:
Please find the next set of comments.
16)
In pg_dump.h, there is a lot of duplication of structures from
conflict.h, we can avoid that by making below changes:
--In SubscriptionInfo(), we can have a list of ConflictTypeResolver
structure and fill the elements of the list in getSubscriptions()
simply by output of pg_subscription_conflict.
--Then in dumpSubscription() we can traverse the list to verify if the
resolver is the default one, if so, skip the dump. We can create a new
function to return whether the resolver is default or not.
--We can get rid of enum ConflictType, enum ConflictResolver,
ConflictResolverNames, ConflictTypeDefaultResolvers from pg_dump.h
17)
In describe.c, we can have an 'order by' in the query so that order is
not changed everytime we update a resolver. Please see this:
For sub1, \dRs was showing below as output for Conflict Resolvers:
insert_exists = error, update_origin_differs = apply_remote,
update_exists = error, update_missing = skip, delete_origin_differs =
apply_remote, delete_missing = skip
Once I update resolver, the order gets changed:
postgres=# ALTER SUBSCRIPTION sub1 CONFLICT RESOLVER
(insert_exists='apply_remote');
ALTER SUBSCRIPTION
\dRs:
update_origin_differs = apply_remote, update_exists = error,
update_missing = skip, delete_origin_differs = apply_remote,
delete_missing = skip, insert_exists = apply_remote
18)
Similarly after making change 16, for pg_dump too, it will be good if
we maintain the order and thus can have order-by in pg_dump's query as
well.
thanks
Shveta
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
Here are some review comments for v14-0001.
This is a WIP, but here are my comments for all the SGML parts.
(There will be some overlap here with comments already posted by Shveta)
======
1. file modes after applying the patchmode change 100644 => 100755 doc/src/sgml/ref/alter_subscription.sgml
mode change 100644 => 100755 doc/src/sgml/ref/create_subscription.sgmlWhat's going on here? Why are those SGMLs changed to executable?
======
Commit message2.
nit - a missing period in the first sentence
nit - typo /reseting/resetting/======
doc/src/sgml/logical-replication.sgml3. - <title>Conflicts</title> + <title>Conflicts and conflict resolution</title>nit - change the capitalisation to "and Conflict Resolution" to match
other titles.~~~
4. + Additional logging is triggered in various conflict scenarios, each identified as a + conflict type, and the conflict statistics are collected (displayed in the + <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view). + Users have the option to configure a <literal>conflict_resolver</literal> for each + <literal>conflict_type</literal> when creating a subscription. + For more information on the supported <literal>conflict_types</literal> detected and + <literal>conflict_resolvers</literal>, refer to section + <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link>. +nit - "Additional logging is triggered" sounds strange. I reworded
this in the nits attachment. Please see if you approve.
nit - The "conflict_type" and "conflict_resolver" you are referring to
here are syntax elements of the CREATE SUBSCRIPTION, so here I think
they should just be called (without the underscores) "conflict type"
and "conflict resolver".
nit - IMO this would be better split into multiple paragraphs.
nit - There is no such section called "CONFLICT RESOLVERS". I reworded
this link text.======
doc/src/sgml/monitoring.sgml5.
The changes here all render with the link including the type "(enum)"
displayed, which I thought it unnecessary/strange.For example:
See insert_exists (enum) for details about this conflict.IIUC there is no problem here, but maybe the other end of the link
needed to define xreflabels. I have made the necessary modifications
in the create_subscription.sgml.======
doc/src/sgml/ref/alter_subscription.sgml6.
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable>
CONFLICT RESOLVER ( <replaceable
class="parameter">conflict_type</replaceable> [= <replaceable
class="parameter">conflict_resolver</replaceable>] [, ...] )This syntax seems wrong to me.
Currently, it says:
ALTER SUBSCRIPTION name CONFLICT RESOLVER ( conflict_type [=
conflict_resolver] [, ...] )But, shouldn't that say:
ALTER SUBSCRIPTION name CONFLICT RESOLVER ( conflict_type =
conflict_resolver [, ...] )~~~
7.
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable>
RESET CONFLICT RESOLVER FOR (<replaceable
class="parameter">conflict_type</replaceable>)I can see that this matches the implementation, but I was wondering
why don't you permit resetting multiple conflict_types at the same
time. e.g. what if I want to reset some but not ALL?~~~
nit - there are some minor whitespace indent problems in the SGML
~~~
8. + <varlistentry id="sql-altersubscription-params-conflict-resolver"> + <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> [= <replaceable class="parameter">conflict_resolver</replaceable>] [, ... ] )</literal></term> + <listitem> + <para> + This clause alters either the default conflict resolvers or those set by <xref linkend="sql-createsubscription"/>. + Refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link> + for the details on supported <literal>conflict_types</literal> and <literal>conflict_resolvers</literal>. + </para> + </listitem> + </varlistentry> + + <varlistentry id="sql-altersubscription-params-conflict-type"> + <term><replaceable class="parameter">conflict_type</replaceable></term> + <listitem> + <para> + The conflict type being reset to its default resolver setting. + For details on conflict types and their default resolvers, refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CONFLICT RESOLVERS</literal></link> + </para> + </listitem> + </varlistentry> + </variablelist>This section seems problematic:
e.g the syntax seems wrong same as before.~
There are other nits.
(I've given a rough fix in the nits attachment. Please see it and make
it better).nit - why do you care if it is "either the default conflict resolvers
or those set...". Why not just say "current resolver"
nit - it does not mention 'conflict_resolver' type in the normal way
nit - there is no actual section called "CONFLICT RESOLVERS"
nit - the part that says "The conflict type being reset to its default
resolver setting." is bogus for this form of the ALTER statement.~~~
9.
There is no description for the "RESET CONFLICT RESOLVER ALL"~~~
10.
There is no description for the "RESET CONFLICT RESOLVER FOR (conflict_type)"======
doc/src/sgml/ref/create_subscription.sgml11. General - Order
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver"> + <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceablenit - IMO this entire new entry about "CONFLICT RESOLVER" should
appear on the page *above* the "WITH" section, because that is the
order that it is defined in the CREATE SUBSCRIPTION syntax.~~~
12. General - whitespace
nit - Much of this new section seems to have a slightly wrong
indentation in the SGML. Mostly it is out by 1 or 2 spaces.~~~
13. General - ordering of conflict_type.
nit - Instead of just some apparent random order, let's put each
insert/update/delete conflict type in alphabetical order, so at least
users can find them where they would expect to find them.
This ordering was decided while implementing the 'conflict-detection
and logging' patch and thus perhaps should be maintained as same. The
ordering is insert, update and delete (different variants of these).
Please see a comment on it in [1]/messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com (comment #2).
[1]: /messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.
I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.
Show quoted text
~~~
15. + <para> + This optional clause specifies options for conflict resolvers for different conflict_types. + </para>nit - IMO we don't need the words "options for" here.
~~~
16. + <para> + The <replaceable class="parameter">conflict_type</replaceable> and their default behaviour are listed below.nit - sounded strange to me. reworded it slightly.
~~~
17.
+ <varlistentry
id="sql-createsubscription-params-with-conflict_type-insert-exists">nit - Here, and for all other conflict types, add "xreflabel". See my
review comment #5 for the reason why.~~~
18. + <para> + The <replaceable class="parameter">conflict_resolver</replaceable> and their behaviour + are listed below. Users can use any of the following resolvers for automatic conflict + resolution. + <variablelist>nit - reworded this too, to be like the previous review comment.
~~~
19. General - readability.
19a.
IMO the information about what are the default resolvers for each
conflict type, and what resolvers are allowed for each conflict type
should ideally be documented in a tabular form.Maybe all information is already present in the current document, but
it is certainly hard to easily see it.As an example, I have added a table in this section. Maybe it is the
best placement for this table, but I gave it mostly how you can
present the same information so it is easier to read.~
19b.
Bug. In doing this exercise I discovered there are 2 resolvers
("error" and "apply_remote") that both claim to be defaults for the
same conflict types.They both say:
+ It is the default resolver for <literal>insert_exists</literal> and + <literal>update_exists</literal>.Anyway, this demonstrates that the current information was hard to read.
I can tell from the code implementation what the document was supposed
to say, but I will leave it to the patch authors to fix this one.
(e.g. "apply_remote" says the wrong defaults)======
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
...
13. General - ordering of conflict_type.
nit - Instead of just some apparent random order, let's put each
insert/update/delete conflict type in alphabetical order, so at least
users can find them where they would expect to find them.This ordering was decided while implementing the 'conflict-detection
and logging' patch and thus perhaps should be maintained as same. The
ordering is insert, update and delete (different variants of these).
Please see a comment on it in [1] (comment #2).[1]:/messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com
+1 for order insert/update/delete.
My issue was only about the order *within* each of those variants.
e.g. I think it should be alphabetical:
CURRENT
insert_exists
update_origin_differs
update_exists
update_missing
delete_origin_differs
delete_missing
SUGGESTED
insert_exists
update_exists
update_missing
update_origin_differs
delete_missing
delete_origin_differs
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.
Resolvers in v14 were documented in this random order:
error
skip
apply_remote
keep_local
apply_or_skip
apply_or_error
Some of these are resolvers for different conflicts. How can you order
these as "resolvers for insert" followed by "resolvers for update"
followed by "resolvers for delete" without it all still appearing in
random order?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
...
13. General - ordering of conflict_type.
nit - Instead of just some apparent random order, let's put each
insert/update/delete conflict type in alphabetical order, so at least
users can find them where they would expect to find them.This ordering was decided while implementing the 'conflict-detection
and logging' patch and thus perhaps should be maintained as same. The
ordering is insert, update and delete (different variants of these).
Please see a comment on it in [1] (comment #2).[1]:/messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com
+1 for order insert/update/delete.
My issue was only about the order *within* each of those variants.
e.g. I think it should be alphabetical:CURRENT
insert_exists
update_origin_differs
update_exists
update_missing
delete_origin_differs
delete_missingSUGGESTED
insert_exists
update_exists
update_missing
update_origin_differs
delete_missing
delete_origin_differs
Okay, got it now. I have no strong opinion here. I am okay with both.
But since it was originally added by other thread, so it will be good
to know the respective author's opinion as well.
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.Resolvers in v14 were documented in this random order:
error
skip
apply_remote
keep_local
apply_or_skip
apply_or_error
Yes, these should be changed.
Some of these are resolvers for different conflicts. How can you order
these as "resolvers for insert" followed by "resolvers for update"
followed by "resolvers for delete" without it all still appearing in
random order?
I was thinking of ordering them like this:
apply_remote: applicable to insert_exists, update_exists,
update_origin_differ, delete_origin_differ
keep_local: applicable to insert_exists,
update_exists, update_origin_differ, delete_origin_differ
apply_or_skip: applicable to update_missing
apply_or_error : applicable to update_missing
skip: applicable to update_missing and
delete_missing
error: applicable to all.
i.e. in order of how they are applicable to conflict_types starting
from insert_exists till delete_origin_differ (i.e. reading
ConflictTypeResolverMap, from left to right and then top to bottom).
Except I have kept 'error' at the end instead of keeping it after
'keep_local' as the former makes more sense there.
thanks
Shveta
On Fri, Sep 27, 2024 at 2:33 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 10:44 AM shveta malik <shveta.malik@gmail.com> wrote:
Thanks for the review.
Here is the v14 patch-set fixing review comments in [1] and [2].Thanks for the patches. I am reviewing patch001, it is WIP, but please
find initial set of comments:
Please find next set of comments:
1)
parse_subscription_conflict_resolvers()
Shall we free 'SeenTypes' list at the end?
2)
General logic comment:
I think SetSubConflictResolver should also accept a list similar to
UpdateSubConflictResolvers() instead of array. Then we can even try
merging these 2 functions later (once we do this change, it will be
more clear). For SetSubConflictResolver to accept a list,
SetDefaultResolvers should give a list as output instead of an array
currently.
3)
Existing logic:
case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
{
ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
/* Remove the existing conflict resolvers. */
RemoveSubscriptionConflictResolvers(subid);
/*
* Create list of conflict resolvers and set them in the
* catalog.
*/
SetDefaultResolvers(conflictResolvers);
SetSubConflictResolver(subid, conflictResolvers, CONFLICT_NUM_TYPES);
}
Suggestion:
If we fix comment #2 and make SetSubConflictResolver and
SetDefaultResolvers to deal with list, then here we can get rid of
RemoveSubscriptionConflictResolvers(), we can simply make a default
list using SetDefaultResolvers and call UpdateSubConflictResolvers().
No need for 2 separate calls for delete and insert/set.
4)
Shall ResetConflictResolver() function also call
UpdateSubConflictResolvers internally? It will get rid of a lot code
duplication.
ResetConflictResolver()'s new approach could be:
a) validate conflict type and get enum value. To do this job, make a
sub-function validate_conflict_type() which will be called both from
here and from validate_conflict_type_and_resolver().
b) get default resolver for given conflict-type enum and then get
resolver string for that to help step c.
c) create a list of single element of ConflictTypeResolver and call
UpdateSubConflictResolvers.
5)
typedefs.list
ConflictResolver is missed?
6)
subscriptioncmds.c
/* Get the list of conflict types and resolvers and validate them. */
conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
No full stop needed in one line comment. But since it is >80 chars,
it is good to split it to multiple lines and then full stop can be
retained.
7)
Shall we move the call to conf_detection_check_prerequisites() to
GetAndValidateSubsConflictResolverList() similar to how we do it for
parse_subscription_conflict_resolvers()? (I still prefer that
GetAndValidateSubsConflictResolverList and
parse_subscription_conflict_resolvers should be merged in the first
place. Array to list conversion as suggested in comment #2 will make
these two functions more similar, and then we can review to merge
them.)
8)
Shall parse_subscription_conflict_resolvers() be moved to conflict.c
as well? Or since it is subscription options' parsing, is it more
suited in the current file? Thoughts?
9)
Existing:
/*
* Parsing function for conflict resolvers in CREATE SUBSCRIPTION command.
* This function will report an error if mutually exclusive or duplicate
* options are specified.
*/
Suggestion:
/*
* Parsing function for conflict resolvers in CREATE SUBSCRIPTION command.
*
* In addition to parsing and validating the resolvers' configuration,
this function
* also reports an error if mutually exclusive options are specified.
*/
10) Test comments (subscription.sql):
------
a)
-- fail - invalid conflict resolvers
CREATE SUBSCRIPTION regress_testsub CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER
(insert_exists = foo) WITH (connect = false);
-- fail - invalid conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION
'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER
(foo = 'keep_local') WITH (connect = false);
We should swap the order of these 2 tests. Make it similar to ALTER tests.
b)
-- fail - invalid conflict resolvers
resolvers-->resolver
-- fail - invalid conflict types
types-->type
-- fail - duplicate conflict types
types->type
c)
-- creating subscription should create default conflict resolvers
Suggestion:
-- creating subscription with no explicit conflict resolvers should
configure default conflict resolvers
d)
-- ok - valid conflict type and resolvers
type-->types
e)
-- fail - altering with duplicate conflict types
types --> type
------
thanks
Shveta
On Mon, Sep 30, 2024 at 4:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.Resolvers in v14 were documented in this random order:
error
skip
apply_remote
keep_local
apply_or_skip
apply_or_errorYes, these should be changed.
Some of these are resolvers for different conflicts. How can you order
these as "resolvers for insert" followed by "resolvers for update"
followed by "resolvers for delete" without it all still appearing in
random order?I was thinking of ordering them like this:
apply_remote: applicable to insert_exists, update_exists,
update_origin_differ, delete_origin_differ
keep_local: applicable to insert_exists,
update_exists, update_origin_differ, delete_origin_differ
apply_or_skip: applicable to update_missing
apply_or_error : applicable to update_missing
skip: applicable to update_missing and
delete_missing
error: applicable to all.i.e. in order of how they are applicable to conflict_types starting
from insert_exists till delete_origin_differ (i.e. reading
ConflictTypeResolverMap, from left to right and then top to bottom).
Except I have kept 'error' at the end instead of keeping it after
'keep_local' as the former makes more sense there.
This proves my point because, without your complicated explanation to
accompany it, the final order (below) just looks random to me:
apply_remote
keep_local
apply_or_skip
apply_or_error
skip
error
Unless there is some compelling reason to do it differently, I still
prefer A-Z (the KISS principle).
======
Kind Regards,
Peter Smith.
Fujitsu Australia
On Mon, Sep 30, 2024 at 2:55 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 4:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.Resolvers in v14 were documented in this random order:
error
skip
apply_remote
keep_local
apply_or_skip
apply_or_errorYes, these should be changed.
Some of these are resolvers for different conflicts. How can you order
these as "resolvers for insert" followed by "resolvers for update"
followed by "resolvers for delete" without it all still appearing in
random order?I was thinking of ordering them like this:
apply_remote: applicable to insert_exists, update_exists,
update_origin_differ, delete_origin_differ
keep_local: applicable to insert_exists,
update_exists, update_origin_differ, delete_origin_differ
apply_or_skip: applicable to update_missing
apply_or_error : applicable to update_missing
skip: applicable to update_missing and
delete_missing
error: applicable to all.i.e. in order of how they are applicable to conflict_types starting
from insert_exists till delete_origin_differ (i.e. reading
ConflictTypeResolverMap, from left to right and then top to bottom).
Except I have kept 'error' at the end instead of keeping it after
'keep_local' as the former makes more sense there.This proves my point because, without your complicated explanation to
accompany it, the final order (below) just looks random to me:
apply_remote
keep_local
apply_or_skip
apply_or_error
skip
errorUnless there is some compelling reason to do it differently, I still
prefer A-Z (the KISS principle).
The "applicable to conflict_types" against each resolver (which will
be mentioned in doc too) is a pretty good reason in itself to keep the
resolvers in the suggested order. To me, it seems more logical than
placing 'apply_or_error' which only applies to the 'update_missing'
conflict_type at the top, while 'error,' which applies to all
conflict_types, placed in the middle. But I understand that
preferences may vary, so I'll leave this to the discretion of others.
thanks
Shveta
On Tue, Oct 1, 2024 at 9:48 AM shveta malik <shveta.malik@gmail.com> wrote:
I have started reviewing patch002, it is WIP, but please find initial
set of comments:
1.
ExecSimpleRelationInsert():
+ /* Check for conflict and return to caller for resolution if found */
+ if (resolver != CR_ERROR &&
+ has_conflicting_tuple(estate, resultRelInfo, &(*conflictslot),
+ CT_INSERT_EXISTS, resolver, slot, subid,
+ apply_remote))
Why are we calling has_conflicting_tuple only if the resolver is not
'ERROR '? Is it for optimization to avoid pre-scan for ERROR cases? If
so, please add a comment.
2)
has_conflicting_tuple():
+ /*
+ * Return if any conflict is found other than one with 'ERROR'
+ * resolver configured. In case of 'ERROR' resolver, emit error here;
+ * otherwise return to caller for resolutions.
+ */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
has_conflicting_tuple() is called only from ExecSimpleRelationInsert()
when the resolver of 'insert_exists' is not 'ERROR', then why do we
have the above comment in has_conflicting_tuple()?
3)
Since has_conflicting_tuple() is only called for insert_exists
conflict, better to name it as 'has_insert_conflicting_tuple' or
'find_insert_conflicting_tuple'. My preference is the second one,
similar to FindConflictTuple().
4)
We can have an ASSERT in has_conflicting_tuple() that conflict_type is
only insert_exists.
5)
has_conflicting_tuple():
+ }
+ return false;
+}
we can have a blank line before returning.
6)
Existing has_conflicting_tuple header comment:
+/*
+ * Check all the unique indexes for conflicts and return true if found.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
Suggestion:
/*
* Check the unique indexes for conflicts. Return true on finding the
first conflict itself.
*
* If the configured resolver is in favour of apply, give the conflicted
* tuple information in conflictslot.
*/
<A change in first line and then a blank line.>
7)
Can we please rearrange 'has_conflicting_tuple' arguments. First
non-pointers, then single pointers and then double pointers.
Oid subid, ConflictType type, ConflictResolver resolver, bool
apply_remote, ResultRelInfo *resultRelInfo, EState *estate,
TupleTableSlot *slot, TupleTableSlot **conflictslot
8)
Now since we are doing a pre-scan of indexes before the actual
table-insert, this existing comment needs some change. Also we need to
mention why we are scanning again when we have done pre-scan already.
/*
* Checks the conflict indexes to fetch the
conflicting local tuple
* and reports the conflict. We perform this check
here, instead of
* performing an additional index scan before the
actual insertion and
* reporting the conflict if any conflicting tuples
are found. This is
* to avoid the overhead of executing the extra scan
for each INSERT
* operation, ....
*/
thanks
Shveta
On Tue, Oct 1, 2024 at 9:54 AM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Oct 1, 2024 at 9:48 AM shveta malik <shveta.malik@gmail.com> wrote:
I have started reviewing patch002, it is WIP, but please find initial
set of comments:
Please find second set of comments for patch002:
9)
can_create_full_tuple():
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
Why are we comparing it with 'LOGICALREP_COLUMN_UNCHANGED'? I assume
toast-values come as LOGICALREP_COLUMN_UNCHANGED. In any case, please
add comments.
10)
There are some alignment changes in
GetAndValidateSubsConflictResolverList() and the next few functions in
the same file which belongs to patch-001. Please move these changes to
patch001.
11)
+ * Find the resolver of the conflict type set under the given subscription.
Suggestion:
Find the resolver for the given conflict type and subscription
12)
+ #include "replication/logicalproto.h"
The code compiles even without the above new inclusion.
13)
ReportApplyConflict:
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, Resolution=%s.",
We can have 'resolution' instead of 'Resolution', similar to
lower-case 'conflict'
14)
errdetail_apply_conflict:
CT_UPDATE_MISSING logs should be improved. As an example:
LOG: conflict detected on relation "public.t1":
conflict=update_missing, Resolution=apply_or_skip.
DETAIL: Could not find the row to be updated, Convert UPDATE to
INSERT and applying the remote changes.
Suggestion:
Could not find the row to be updated, thus converting the UPDATE to an
INSERT and applying the remote changes.
Similarly for other lines:
Could not find the row to be updated, and the UPDATE cannot be
converted to an INSERT, thus skipping the remote changes.
Could not find the row to be updated, and the UPDATE cannot be
converted to an INSERT, thus raising the error.
15)
errdetail_apply_conflict:
Can we pull out the sentence 'Could not find the row to be updated',
as it is common for all the cases of 'CT_UPDATE_MISSING' and then
append the rest of the string to it case-wise?
16)
+ConflictResolver
+GetConflictResolver(Relation localrel, ConflictType type, bool *apply_remote,
+ LogicalRepTupleData *newtup, Oid subid)
Can we please change the order of args to:
Oid subid, ConflictType type, Relation localrel, LogicalRepTupleData
*newtup, bool *apply_remote
Since we are getting resolvers for 'subid' and 'type', I have kept
those as initial args and OUT argument as last one.
17)
apply_handle_insert_internal:
+ /*
+ * If a conflict is detected and resolver is in favor of applying the
+ * remote changes, update the conflicting tuple by converting the remote
+ * INSERT to an UPDATE.
+ */
+ if (conflictslot)
The comment conveys 2 conditions while the code checks only one
condition, thus it is slightly misleading.
Perhaps change comment to:
/*
* If a conflict is detected, update the conflicting tuple by
converting the remote
* INSERT to an UPDATE. Note that conflictslot will have the
conflicting tuple only if
* the resolver is in favor of applying the changes, otherwise it will be NULL.
*/
<Rephrase if needed>
18)
apply_handle_update_internal():
* Report the conflict and configured resolver if the tuple was
Remove extra space after conflict.
thanks
Shveta
Hi, here are my code/test review comments for patch v14-0001.
(the v14-0001 docs review was already previously posted)
======
src/backend/commands/subscriptioncmds.c
parse_subscription_conflict_resolvers:
1.
+ /* Check if the conflict type already exists in the list */
+ if (list_member(SeenTypes, makeString(defel->defname)))
This 'SeenTypes' logic of string comparison of list elements all seems
overkill to me.
There is a known number of conflict types, so why not use a more
efficient bool array:
e.g. bool seen_conflict_type[CONFLICT_NUM_TYPES] = {0};
NOTE - I've already made this change in the nits attachment to
demonstrate that it works fine.
~
2.
+ foreach(lc, stmtresolvers)
+ {
+ DefElem *defel = (DefElem *) lfirst(lc);
There are macros to do this, so you don't need to use lfirst(). e.g.
foreach_ptr?
~
3.
nit (a) - remove excessive blank lines
nit (b) - minor comment reword + periods.
~~~
CreateSubscription:
4.
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
nit - Suggest rename to 'conflict_resolvers' to keep similar style to
other variables.
~
5.
nit - add a missing period to some comments.
~~~
AlterSubscription:
6.
nit (a) - remove excessing blank lines.
nit (b) - add missing period in comments.
nit (c) - rename 'conflictResolvers' to 'conflict_resolvers' for
consistent variable style
======
src/backend/parser/gram.y
7.
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
+ n->subname = $3;
+ n->conflict_type = $8;
+ $$ = (Node *) n;
+ }
Why does this only support resetting resolvers for one conflict type at a time?
~
8.
nit - add whitespace blank line
======
src/backend/replication/logical/conflict.c
9.
Add some static assertions for all of the arrays to ensure everything
is declared as expected.
(I've made this change in the nit attachment)
~~~
10.
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
It's a bit fiddly introducing this constant just for the map
dimensions. You might as well use the already defined
CONFLICT_NUM_RESOLVERS.
Yes, I know they are not exactly the same. But, the extra few ints of
space wasted reusing this is insignificant, and there is no
performance loss in the lookup logic (after my other review comment
later). IMO it is easier to use what you already have instead of
introducing yet another hard-to-explain constant.
(I've made this change in the nit attachment)
~~~
11.
+static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
+ [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
+ [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP,
CR_ERROR},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+};
FYI - There is a subtle quirk (and potential bug) lurking in the way
this map is declared. The array elements for resolvers that are not
defined will default to value 0. So "{CR_SKIP, CR_ERROR}" is
essentially saying "{CR_SKIP, CR_ERROR, 0, 0}", and that means that
the enums for resolvers cannot have a value 0 because then this
ConflictTypeResolverMap would be broken. I see that this was already
handled (enums started at 1) but there was no explanatory comment so
it would be easy to unknowingly break it.
A better way (what I am suggesting here) would be to have a -1 marker
to indicate the end of the list of valid resolvers. This removes any
doubt, and it means the enums can start from 0 as normal, and it means
you can quick-exit from the lookup code (suggested later below) for a
performance gain. Basically, win-win-win.
For example:
* NOTE: -1 is an end marker for the list of valid resolvers for each conflict
* type.
*/
static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS+1] = {
[CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
[CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
[CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
[CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP,
CR_ERROR, -1},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
[CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
};
(I have made this change already in the nit attachment)
~~~
12.
+/*
+ * Default conflict resolver for each conflict type.
+ */
+static const int ConflictTypeDefaultResolvers[] = {
+ [CT_INSERT_EXISTS] = CR_ERROR,
+ [CT_UPDATE_EXISTS] = CR_ERROR,
+ [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
+ [CT_UPDATE_MISSING] = CR_SKIP,
+ [CT_DELETE_MISSING] = CR_SKIP,
+ [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
+
+};
+
IMO you can throw all this away. Instead, simply rearrange the
resolvers in ConflictTypeResolverMap[] and provide a comment to say
that the resolver at index [0] of the valid resolver lists is the
"default" resolver for each conflict type.
(I have made this change already in the nit attachment)
~~~
SetDefaultResolvers:
13.
+/*
+ * Set default values for CONFLICT RESOLVERS for each conflict type
+ */
+void
+SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+{
+ ConflictType type;
+
+ for (type = CT_MIN; type <= CT_MAX; type++)
+ {
+ conflictResolvers[type].conflict_type = ConflictTypeNames[type];
+ conflictResolvers[type].resolver =
+ ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ }
+}
nit (a) - remove extra blank line about this function
nit (b) - add a period to the function comment
nit (c) - tidy the param.
nit (d) - use for-loop variable
nit (e) - don't need CT_MIN and CT_MAX
~~~
validate_conflict_type_and_resolver:
14.
AFAIK, you could also break out of the resolver loop early if you hit
an invalid resolver, because that means you have already fallen off
the end of the valid resolvers.
e.g. do this (in conjunction with the other suggested
ConflictTypeResolverMap changes)
if (candidate < 0)
{
/* No more possible resolvers for this conflict type. */
break;
}
~
15.
nit (a) - use for-loop variable
nit (b) - improve the comments a bit
nit (c) - don't need CT_MIN and CT_MAX
nit (d) - don't need CR_MIN and CR_MAX
nit (e) - use loop variable 'i'
nit (f) - add a blank line before the return
nit (g) - remove excessive blank lines in the function (after the return)
~~~
GetAndValidateSubsConflictResolverList:
16.
nit (a) - minor tweak function comment
nit (b) - put the "ConflictTypeResolver *conftyperesolver = NULL" in
lower scope.
nit (c) - tweak comments for periods etc.
~
17.
+ /* Add the conflict type to the list of seen types */
+ conflictTypes = lappend(conflictTypes,
+ makeString((char *) conftyperesolver->conflict_type));
All this messing with lists of strings seems somewhat inefficient. We
already know the enum by this point (e.g.
validate_conflict_type_and_resolver returns it) so the "seen" logic
should be possible with a simple bool array.
(I've made this change in the nits attachment to give an example of
what I mean. It seems to work just fine).
~~~
UpdateSubConflictResolvers:
18.
+ if (HeapTupleIsValid(oldtup))
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confrres - 1] =
+ CStringGetTextDatum(conftyperesolver->resolver);
+ replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict, &oldtup->t_self, newtup);
+ ReleaseSysCache(oldtup);
+ heap_freetuple(newtup);
+ }
+ else
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type, subid);
I think this might be more easily expressed by reversing the condition:
if (!HeapTupleIsValid(oldtup))
elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
conftyperesolver->conflict_type, subid);
...
~~~
ResetConflictResolver:
19.
+ResetConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver conflictResolver;
No point in declaring the variable as a ConflictTypeResolver because
the code is only interested in the resolver name field.
~
20.
nit (a) - Add period to code comments
nit (b) - Remove excess blank lines
nit (c) - Change 'conflict_type' to the words 'conflict type' in a
couple of comments
~~~
conf_detection_check_prerequisites:
21.
Why do we get this warning repeated when just setting the same
resolver as the conflict already has? Even overwriting the resolver
default does this.
~~~
SetSubConflictResolver:
22.
nit (a) - Add period to function comment.
nit (b) - Tidy the spaces in the param declaration
nit (c) - Add period in code comments.
~~~
RemoveSubscriptionConflictResolvers:
23.
nit (a) - Add period to function comment.
nit (b) - Minor change to comment wording.
nit (c) - Add period code to comment.
======
src/bin/pg_dump/pg_dump.c
SKIP REVIEW
======
src/bin/pg_dump/pg_dump.h
SKIP REVIEW
======
src/bin/psql/describe.c
24.
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", (SELECT string_agg(confrtype || ' = ' || confrres, ', ') \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " WHERE c.confsubid = s.oid) AS \"%s\"\n",
+ gettext_noop("Conflict Resolvers"));
Should this SQL include an ORDER BY so the result is the same each time?
~~~
25. Present the information separately.
The current result is a lot of text, which is not particularly
readable. Even when using the "expanded display" mode.
Name | sub1
Owner | postgres
Enabled | t
Publication | {pub1}
Binary | f
Streaming | off
Two-phase commit | d
Disable on error | f
Origin | any
Password required | t
Run as owner? | f
Failover | f
Synchronous commit | off
Conninfo | dbname=test_pub
Skip LSN | 0/0
Conflict Resolvers | insert_exists = error, update_origin_differs =
apply_remote, update_exists = error, update_missing = skip,
delete_origin_differs = apply_remote, delete_missing = skip
I am thinking perhaps the "Conflict Resolver" information should be
presented separately in a list *below* the normal "describe" output.
Other kinds of describe present information this way too (e.g. the
publication lists tables below).
~
26. Identify what was changed by the user
IMO it may also be useful to know which of the conflict resolvers are
the defaults anyway, versus which have been changed to something else
by the user. If you present the information listed separately like
suggested (above) then there would be room to add this additional
info.
======
.../catalog/pg_subscription_conflict.h
27.
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text confrtype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confrres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
These seem strange field names. What do that mean? e.g. for
'confrtype' what is the 'r'?
======
src/include/nodes/parsenodes.h
28.
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
That 'resolvers' comment should clarify what kind of a list that is.
e.g. Conflict type enums or strings etc.
~~~
29.
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER,
} AlterSubscriptionType;
I think these names should mimic the actual syntax more closely.
e.g.
ALTER_SUBSCRIPTION_CONFLICT_RESOLVER,
ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET
ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL
~~~
30.
+ List *resolvers; /* List of conflict resolvers */
+ char *conflict_type; /* conflict_type to be reset */
} AlterSubscriptionStmt;
30a.
(same as earlier review comment #28)
That 'resolvers' comment should clarify what kind of a list that is.
e.g. Conflict type enums or strings etc.
~
30b.
nit (a) - better name here might be "conflict_type_name", so it is not
confused with the enum
nit (b) - fix the comment: e.g. you can't define 'conflict_type' in
terms of 'conflict_type'
======
src/include/replication/conflict.h
31.
+/* Min and max conflict type */
+#define CT_MIN CT_INSERT_EXISTS
+#define CT_MAX CT_DELETE_MISSING
+
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
nit - these CT_MIN and CT_MAX are unnecessary. If the enums start at 0
(which they do in C by default anyhow) then you only need to know
'CONFLICT_NUM_TYPES'. Code previously using CT_MIN and CT_MAX will
become simpler too.
~~~
32.
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE = 1,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it can not be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it can not be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+/* Min conflict resolver */
+#define CR_MIN CR_APPLY_REMOTE
+#define CR_MAX CR_ERROR
+
nit (a) - /can not/cannot/
nit (b) - as above, those CR_MIN and CR_MAX are not necessary. It is
simpler to start the enum values at 0 and then you only need to know
CONFLICT_NUM_RESOLVERS. Code previously using CT_MIN and CT_MAX will
become simpler too.
(I made these changes already -- see the attachment).
~~~
33.
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
These seem poor field names that are too easily confused with the
enums etc. Let's name them to make it obvious what they really are:
/conflict_type/conflict_type_name/
/resolver/conflict_resolver_name/
~~~
34.
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver *
resolvers, int max_types);
+extern void RemoveSubscriptionConflictById(Oid confid);
+extern void RemoveSubscriptionConflictResolvers(Oid confid);
+extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType validate_conflict_type_and_resolver(const char
*conflict_type,
+ const char *conflict_resolver);
+extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void ResetConflictResolver(Oid subid, char *conflict_type);
+extern void conf_detection_check_prerequisites(void);
nit (a) - remove some spaces that should not be there
nit (b) - make the parameter names more consistent
======
src/test/regress/sql/subscription.sql
35. General - quotes?
Why are all the resolver names single-quoted? (e.g. "update_missing =
'skip'"). AFAIK it is unnecessary.
~
36. General - why not use describe?
Would the text code be easier if you just validated the changes by
using the "describe" code (e.g. \dRs+ regress_testsub) instead of SQL
select of the pg_subscription_conflict table?
~
37.
nit - The test cases seemed mostly good to me. My nits are only for
small changes/typos or clarifications to the comments.
======
Please see the PS nits attachment. It already implements most of the
nits plus some other review comments were suggested above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
PS_NITPICKS_20241002_v140001_CODE.txttext/plain; charset=US-ASCII; name=PS_NITPICKS_20241002_v140001_CODE.txtDownload
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 859bf08..703d313 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -450,10 +450,9 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
ConflictTypeResolver *resolvers)
{
ListCell *lc;
- List *SeenTypes = NIL;
+ bool already_seen[CONFLICT_NUM_TYPES] = {0};
-
- /* First initialize the resolvers with default values. */
+ /* First, initialize the resolvers with default values. */
SetDefaultResolvers(resolvers);
foreach(lc, stmtresolvers)
@@ -462,28 +461,26 @@ parse_subscription_conflict_resolvers(List *stmtresolvers,
ConflictType type;
char *resolver;
- /* Check if the conflict type already exists in the list */
- if (list_member(SeenTypes, makeString(defel->defname)))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("duplicate conflict type \"%s\" found", defel->defname)));
-
/* Validate the conflict type and resolver. */
resolver = defGetString(defel);
type = validate_conflict_type_and_resolver(defel->defname,
defGetString(defel));
- /* Add the conflict type to the list of seen types */
- SeenTypes = lappend(SeenTypes, makeString((char *)resolvers[type].conflict_type));
+ /* Check if the conflict type has been already seen. */
+ if (already_seen[type])
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", defel->defname)));
+
+ already_seen[type] = true;
/* Update the corresponding resolver for the given conflict type. */
- resolvers[type].resolver = downcase_truncate_identifier(resolver, strlen(resolver), false);
+ resolvers[type].conflict_resolver_name = downcase_truncate_identifier(resolver, strlen(resolver), false);
}
/* Once validation is complete, warn users if prerequisites are not met. */
if (stmtresolvers)
conf_detection_check_prerequisites();
-
}
/*
@@ -630,7 +627,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
- ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
+ ConflictTypeResolver conflict_resolvers[CONFLICT_NUM_TYPES];
/*
* Parse and check options.
@@ -646,7 +643,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
/* Parse and check conflict resolvers. */
- parse_subscription_conflict_resolvers(stmt->resolvers, conflictResolvers);
+ parse_subscription_conflict_resolvers(stmt->resolvers, conflict_resolvers);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -774,8 +771,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
- /* Update the Conflict Resolvers in pg_subscription_conflict */
- SetSubConflictResolver(subid, conflictResolvers, CONFLICT_NUM_TYPES);
+ /* Update the Conflict Resolvers in pg_subscription_conflict. */
+ SetSubConflictResolver(subid, conflict_resolvers, CONFLICT_NUM_TYPES);
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1639,11 +1636,10 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
{
List *conflict_resolvers = NIL;
-
/* Get the list of conflict types and resolvers and validate them. */
conflict_resolvers = GetAndValidateSubsConflictResolverList(stmt->resolvers);
- /* Warn users if prerequisites are not met */
+ /* Warn users if prerequisites are not met. */
conf_detection_check_prerequisites();
/*
@@ -1655,7 +1651,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
}
case ALTER_SUBSCRIPTION_RESET_ALL_CONFLICT_RESOLVERS:
{
- ConflictTypeResolver conflictResolvers[CONFLICT_NUM_TYPES];
+ ConflictTypeResolver conflict_resolvers[CONFLICT_NUM_TYPES];
/* Remove the existing conflict resolvers. */
RemoveSubscriptionConflictResolvers(subid);
@@ -1664,8 +1660,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
* Create list of conflict resolvers and set them in the
* catalog.
*/
- SetDefaultResolvers(conflictResolvers);
- SetSubConflictResolver(subid, conflictResolvers, CONFLICT_NUM_TYPES);
+ SetDefaultResolvers(conflict_resolvers);
+ SetSubConflictResolver(subid, conflict_resolvers, CONFLICT_NUM_TYPES);
break;
}
case ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER:
@@ -1674,7 +1670,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
* Reset the conflict resolver for this conflict type to its
* default.
*/
- ResetConflictResolver(subid, stmt->conflict_type);
+ ResetConflictResolver(subid, stmt->conflict_type_name);
break;
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 8b7023d..e94c91f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -10870,10 +10870,11 @@ AlterSubscriptionStmt:
n->kind = ALTER_SUBSCRIPTION_RESET_CONFLICT_RESOLVER;
n->subname = $3;
- n->conflict_type = $8;
+ n->conflict_type_name = $8;
$$ = (Node *) n;
}
;
+
conflict_type:
Sconst { $$ = $1; }
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index ff44c3c..a75f478 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -40,6 +40,9 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+StaticAssertDecl(lengthof(ConflictTypeNames) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
+
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
@@ -49,7 +52,8 @@ static const char *const ConflictResolverNames[] = {
[CR_ERROR] = "error"
};
-#define CONFLICT_TYPE_MAX_RESOLVERS 4
+StaticAssertDecl(lengthof(ConflictResolverNames) == CONFLICT_NUM_RESOLVERS,
+ "array length mismatch");
/*
* Valid conflict resolvers for each conflict type.
@@ -66,28 +70,23 @@ static const char *const ConflictResolverNames[] = {
* Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
* and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
* friendly name for a resolver and thus has been added here.
+ *
+ * NOTES:
+ * The first resolver (i.e. resolver at index [0]) is the default
+ * resolver for each conflict type. There is a -1 end marker for each list
+ * valid resolvers.
*/
-static const int ConflictTypeResolverMap[][CONFLICT_TYPE_MAX_RESOLVERS] = {
- [CT_INSERT_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_EXISTS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR},
- [CT_UPDATE_MISSING] = {CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_SKIP, CR_ERROR},
- [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR}
+static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS+1] = {
+ [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
+ [CT_UPDATE_MISSING] = {CR_SKIP, CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_ERROR, -1},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
};
-/*
- * Default conflict resolver for each conflict type.
- */
-static const int ConflictTypeDefaultResolvers[] = {
- [CT_INSERT_EXISTS] = CR_ERROR,
- [CT_UPDATE_EXISTS] = CR_ERROR,
- [CT_UPDATE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE,
- [CT_UPDATE_MISSING] = CR_SKIP,
- [CT_DELETE_MISSING] = CR_SKIP,
- [CT_DELETE_ORIGIN_DIFFERS] = CR_APPLY_REMOTE
-
-};
+StaticAssertDecl(lengthof(ConflictTypeResolverMap) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
@@ -547,20 +546,18 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
-
/*
- * Set default values for CONFLICT RESOLVERS for each conflict type
+ * Set default values for CONFLICT RESOLVERS for each conflict type.
*/
void
-SetDefaultResolvers(ConflictTypeResolver * conflictResolvers)
+SetDefaultResolvers(ConflictTypeResolver *resolvers)
{
- ConflictType type;
-
- for (type = CT_MIN; type <= CT_MAX; type++)
+ for (ConflictType type = 0; type < CONFLICT_NUM_TYPES; type++)
{
- conflictResolvers[type].conflict_type = ConflictTypeNames[type];
- conflictResolvers[type].resolver =
- ConflictResolverNames[ConflictTypeDefaultResolvers[type]];
+ ConflictResolver def_resolver = ConflictTypeResolverMap[type][0];
+
+ resolvers[type].conflict_type_name = ConflictTypeNames[type];
+ resolvers[type].conflict_resolver_name = ConflictResolverNames[def_resolver];
}
}
@@ -575,10 +572,9 @@ validate_conflict_type_and_resolver(const char *conflict_type,
ConflictType type;
ConflictResolver resolver;
bool valid = false;
- int i;
- /* Check conflict type validity */
- for (type = CT_MIN; type <= CT_MAX; type++)
+ /* Validate the conflict type name. */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
{
if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
{
@@ -595,8 +591,8 @@ validate_conflict_type_and_resolver(const char *conflict_type,
/* Reset */
valid = false;
- /* Check conflict resolver validity. */
- for (resolver = CR_MIN; resolver <= CR_MAX; resolver++)
+ /* Validate the conflict resolver name. */
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
{
if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
{
@@ -614,9 +610,17 @@ validate_conflict_type_and_resolver(const char *conflict_type,
valid = false;
/* Check if conflict resolver is a valid one for the given conflict type */
- for (i = 0; i < CONFLICT_TYPE_MAX_RESOLVERS; i++)
+ for (int i = 0; i < CONFLICT_NUM_RESOLVERS; i++)
{
- if (ConflictTypeResolverMap[type][i] == resolver)
+ int candidate = ConflictTypeResolverMap[type][i];
+
+ if (candidate < 0)
+ {
+ /* No more possible resolvers for this conflict type. */
+ break;
+ }
+
+ if (candidate == resolver)
{
valid = true;
break;
@@ -629,51 +633,48 @@ validate_conflict_type_and_resolver(const char *conflict_type,
errmsg("%s is not a valid conflict resolver for conflict type %s",
conflict_resolver,
conflict_type));
- return type;
+ return type;
}
/*
- * Extract the conflict type and conflict resolvers from the
+ * Extract the conflict types and conflict resolvers from the
* ALTER SUBSCRIPTION command and return a list of ConflictTypeResolver nodes.
*/
List *
GetAndValidateSubsConflictResolverList(List *stmtresolvers)
{
- ConflictTypeResolver *conftyperesolver = NULL;
List *res = NIL;
- List *conflictTypes = NIL;
+ bool already_seen[CONFLICT_NUM_TYPES] = {0};
foreach_ptr(DefElem, defel, stmtresolvers)
{
+ ConflictType conflict_type;
+ ConflictTypeResolver *conftyperesolver;
char *resolver_str;
- /* Check if the conflict type already exists in the list */
- if (list_member(conflictTypes, makeString(defel->defname)))
- {
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("duplicate conflict type \"%s\" found", defel->defname)));
- }
-
conftyperesolver = palloc(sizeof(ConflictTypeResolver));
- conftyperesolver->conflict_type = downcase_truncate_identifier(defel->defname,
+ conftyperesolver->conflict_type_name = downcase_truncate_identifier(defel->defname,
strlen(defel->defname), false);
resolver_str = defGetString(defel);
- conftyperesolver->resolver = downcase_truncate_identifier(resolver_str,
+ conftyperesolver->conflict_resolver_name = downcase_truncate_identifier(resolver_str,
strlen(resolver_str), false);
/*
- * Validate the conflict type and that the resolver is valid for that
- * conflict type
+ * Validate the conflict type, and check the resolver is valid for that
+ * conflict type.
*/
- validate_conflict_type_and_resolver(
- conftyperesolver->conflict_type,
- conftyperesolver->resolver);
+ conflict_type = validate_conflict_type_and_resolver(
+ conftyperesolver->conflict_type_name,
+ conftyperesolver->conflict_resolver_name);
+
+ /* Check if the conflict type has been seen already. */
+ if (already_seen[conflict_type])
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("duplicate conflict type \"%s\" found", conftyperesolver->conflict_type_name)));
- /* Add the conflict type to the list of seen types */
- conflictTypes = lappend(conflictTypes,
- makeString((char *) conftyperesolver->conflict_type));
+ already_seen[conflict_type] = true;
/* Add the validated ConflictTypeResolver to the result list */
res = lappend(res, conftyperesolver);
@@ -711,7 +712,7 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
/* set up subid and conflict_type to search in cache */
values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_conflict_confrtype - 1] =
- CStringGetTextDatum(conftyperesolver->conflict_type);
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
values[Anum_pg_subscription_conflict_confsubid - 1],
@@ -721,7 +722,7 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
{
/* Update the new resolver */
values[Anum_pg_subscription_conflict_confrres - 1] =
- CStringGetTextDatum(conftyperesolver->resolver);
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
@@ -732,7 +733,7 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
}
else
elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
- conftyperesolver->conflict_type, subid);
+ conftyperesolver->conflict_type_name, subid);
}
@@ -746,7 +747,7 @@ void
ResetConflictResolver(Oid subid, char *conflict_type)
{
ConflictType idx;
- ConflictTypeResolver conflictResolver;
+ const char *resolver_name;
Datum values[Natts_pg_subscription_conflict];
bool nulls[Natts_pg_subscription_conflict];
bool replaces[Natts_pg_subscription_conflict];
@@ -757,8 +758,8 @@ ResetConflictResolver(Oid subid, char *conflict_type)
char *cur_conflict_res;
Datum datum;
- /* Get the index for this conflict_type */
- for (idx = CT_MIN; idx <= CT_MAX; idx++)
+ /* Get the index for this conflict type. */
+ for (idx = 0; idx < CONFLICT_NUM_TYPES; idx++)
{
if (pg_strcasecmp(ConflictTypeNames[idx], conflict_type) == 0)
{
@@ -772,8 +773,8 @@ ResetConflictResolver(Oid subid, char *conflict_type)
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("%s is not a valid conflict type", conflict_type));
- /* Get the default resolver for this conflict_type. */
- conflictResolver.resolver = ConflictResolverNames[ConflictTypeDefaultResolvers[idx]];
+ /* Get the default resolver for this conflict type. */
+ resolver_name = ConflictResolverNames[ConflictTypeResolverMap[idx][0]];
/* Prepare to update a tuple. */
memset(nulls, false, sizeof(nulls));
@@ -789,7 +790,6 @@ ResetConflictResolver(Oid subid, char *conflict_type)
values[Anum_pg_subscription_conflict_confsubid - 1],
values[Anum_pg_subscription_conflict_confrtype - 1]);
-
if (!HeapTupleIsValid(oldtup))
elog(ERROR, "cache lookup failed for table conflict %s for subid %u", conflict_type, subid);
@@ -798,11 +798,11 @@ ResetConflictResolver(Oid subid, char *conflict_type)
cur_conflict_res = TextDatumGetCString(datum);
/* Check if current resolver is the default one, if not update it. */
- if (pg_strcasecmp(cur_conflict_res, conflictResolver.resolver) != 0)
+ if (pg_strcasecmp(cur_conflict_res, resolver_name) != 0)
{
- /* Update the new resolver */
+ /* Update the new resolver. */
values[Anum_pg_subscription_conflict_confrres - 1] =
- CStringGetTextDatum(conflictResolver.resolver);
+ CStringGetTextDatum(resolver_name);
replaces[Anum_pg_subscription_conflict_confrres - 1] = true;
newtup = heap_modify_tuple(oldtup, RelationGetDescr(pg_subscription_conflict),
@@ -832,10 +832,10 @@ conf_detection_check_prerequisites(void)
}
/*
- * Set Conflict Resolvers on the subscription
+ * Set Conflict Resolvers on the subscription.
*/
void
-SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolvers_cnt)
+SetSubConflictResolver(Oid subId, ConflictTypeResolver *resolvers, int resolvers_cnt)
{
Relation pg_subscription_conflict;
Datum values[Natts_pg_subscription_conflict];
@@ -853,11 +853,11 @@ SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolver
{
values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subId);
values[Anum_pg_subscription_conflict_confrtype - 1] =
- CStringGetTextDatum(resolvers[idx].conflict_type);
+ CStringGetTextDatum(resolvers[idx].conflict_type_name);
values[Anum_pg_subscription_conflict_confrres - 1] =
- CStringGetTextDatum(resolvers[idx].resolver);
+ CStringGetTextDatum(resolvers[idx].conflict_resolver_name);
- /* Get a new oid and update the tuple into catalog */
+ /* Get a new oid and update the tuple into catalog. */
conflict_oid = GetNewOidWithIndex(pg_subscription_conflict, SubscriptionConflictOidIndexId,
Anum_pg_subscription_conflict_oid);
values[Anum_pg_subscription_conflict_oid - 1] = ObjectIdGetDatum(conflict_oid);
@@ -871,7 +871,7 @@ SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int resolver
}
/*
- * Remove the subscription conflict resolvers for the subscription id
+ * Remove the subscription conflict resolvers for the subscription id.
*/
void
RemoveSubscriptionConflictResolvers(Oid subid)
@@ -884,8 +884,8 @@ RemoveSubscriptionConflictResolvers(Oid subid)
rel = table_open(SubscriptionConflictId, RowExclusiveLock);
/*
- * Search using the subid, this should return all conflict resolvers for
- * this sub
+ * Search using the subid to return all conflict resolvers for
+ * this subscription.
*/
ScanKeyInit(&skey[0],
Anum_pg_subscription_conflict_confsubid,
@@ -895,7 +895,7 @@ RemoveSubscriptionConflictResolvers(Oid subid)
scan = table_beginscan_catalog(rel, 1, skey);
- /* Iterate through the tuples and delete them */
+ /* Iterate through the tuples and delete them. */
while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7e19e0c..0b2832d 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4228,7 +4228,7 @@ typedef struct AlterSubscriptionStmt
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
List *resolvers; /* List of conflict resolvers */
- char *conflict_type; /* conflict_type to be reset */
+ char *conflict_type_name; /* Name of the conflict type to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index fcd49da..c5c3c20 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -50,10 +50,6 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
-/* Min and max conflict type */
-#define CT_MIN CT_INSERT_EXISTS
-#define CT_MAX CT_DELETE_MISSING
-
/*
* Conflict resolvers that can be used to resolve various conflicts.
*
@@ -63,15 +59,15 @@ typedef enum
typedef enum ConflictResolver
{
/* Apply the remote change */
- CR_APPLY_REMOTE = 1,
+ CR_APPLY_REMOTE,
/* Keep the local change */
CR_KEEP_LOCAL,
- /* Apply the remote change; skip if it can not be applied */
+ /* Apply the remote change; skip if it cannot be applied */
CR_APPLY_OR_SKIP,
- /* Apply the remote change; emit error if it can not be applied */
+ /* Apply the remote change; emit error if it cannot be applied */
CR_APPLY_OR_ERROR,
/* Skip applying the change */
@@ -81,14 +77,12 @@ typedef enum ConflictResolver
CR_ERROR,
} ConflictResolver;
-/* Min conflict resolver */
-#define CR_MIN CR_APPLY_REMOTE
-#define CR_MAX CR_ERROR
+#define CONFLICT_NUM_RESOLVERS (CR_ERROR + 1)
typedef struct ConflictTypeResolver
{
- const char *conflict_type;
- const char *resolver;
+ const char *conflict_type_name;
+ const char *conflict_resolver_name;
} ConflictTypeResolver;
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
@@ -103,14 +97,14 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
-extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver * resolvers, int max_types);
+extern void SetSubConflictResolver(Oid subId, ConflictTypeResolver *resolvers, int max_types);
extern void RemoveSubscriptionConflictById(Oid confid);
extern void RemoveSubscriptionConflictResolvers(Oid confid);
extern List *GetAndValidateSubsConflictResolverList(List *stmtresolvers);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType validate_conflict_type_and_resolver(const char *conflict_type,
const char *conflict_resolver);
-extern void SetDefaultResolvers(ConflictTypeResolver * conflictResolvers);
+extern void SetDefaultResolvers(ConflictTypeResolver *resolvers);
extern void ResetConflictResolver(Oid subid, char *conflict_type);
extern void conf_detection_check_prerequisites(void);
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 9fa2a3f..c62c745 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -407,14 +407,14 @@ ERROR: foo is not a valid conflict resolver
-- fail - invalid conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
ERROR: foo is not a valid conflict type
--- fail - invalid resolver for that conflict type
+-- fail - invalid resolver for given conflict type
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
-- fail - duplicate conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
ERROR: duplicate conflict type "insert_exists" found
--- creating subscription should create default conflict resolvers
+-- ok - create subscription with no conflict resolvers should create defaults
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
@@ -431,13 +431,13 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--- ok - valid conflict type and resolvers
+-- ok - create subscription specifying valid conflict type and resolvers
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
---check if above are configured; for non specified conflict types, default resolvers should be seen
+--check if above are configured; for non-specified conflict types, default resolvers should be seen
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
confrtype | confrres
-----------------------+--------------
@@ -449,16 +449,16 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
update_origin_differs | apply_remote
(6 rows)
--- fail - altering with invalid conflict type
+-- fail - alter with invalid conflict type
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
ERROR: foo is not a valid conflict type
--- fail - altering with invalid conflict resolver
+-- fail - alter with invalid conflict resolver
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
ERROR: foo is not a valid conflict resolver
--- fail - altering with duplicate conflict types
+-- fail - alter with duplicate conflict types
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
ERROR: duplicate conflict type "insert_exists" found
--- ok - valid conflict types and resolvers
+-- ok - alter to set valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
@@ -473,7 +473,7 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
update_origin_differs | apply_remote
(6 rows)
--- ok - valid conflict types and resolvers
+-- ok - alter to set valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
@@ -488,10 +488,10 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
update_origin_differs | error
(6 rows)
--- fail - reset with an invalid conflit type
+-- fail - alter to reset an invalid conflict type
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
ERROR: foo is not a valid conflict type
--- ok - valid conflict type
+-- ok - alter to reset a valid conflict type
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
confrtype | confrres
@@ -504,7 +504,7 @@ SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
update_origin_differs | error
(6 rows)
--- ok - reset ALL
+-- ok - alter to reset conflict resolvers for all conflict types
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
confrtype | confrres
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 596f2e1..b081c1d 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -278,51 +278,50 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
-- fail - invalid conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
--- fail - invalid resolver for that conflict type
+-- fail - invalid resolver for given conflict type
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
-- fail - duplicate conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
--- creating subscription should create default conflict resolvers
+-- ok - create subscription with no conflict resolvers should create defaults
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
--- ok - valid conflict type and resolvers
+-- ok - create subscription specifying valid conflict type and resolvers
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
-
---check if above are configured; for non specified conflict types, default resolvers should be seen
+--check if above are configured; for non-specified conflict types, default resolvers should be seen
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
--- fail - altering with invalid conflict type
+-- fail - alter with invalid conflict type
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
--- fail - altering with invalid conflict resolver
+-- fail - alter with invalid conflict resolver
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
--- fail - altering with duplicate conflict types
+-- fail - alter with duplicate conflict types
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
--- ok - valid conflict types and resolvers
+-- ok - alter to set valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
--- ok - valid conflict types and resolvers
+-- ok - alter to set valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
--- fail - reset with an invalid conflit type
+-- fail - alter to reset an invalid conflict type
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
--- ok - valid conflict type
+-- ok - alter to reset a valid conflict type
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
--- ok - reset ALL
+-- ok - alter to reset conflict resolvers for all conflict types
ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
SELECT confrtype, confrres FROM pg_subscription_conflict ORDER BY confrtype;
On Wed, Oct 2, 2024 at 6:55 PM Peter Smith <smithpb2250@gmail.com> wrote:
Hi, here are my code/test review comments for patch v14-0001.
Here is v15 patch (patch-0001), addressing comments from [1]/messages/by-id/CAJpy0uBS1_wKzmhP9uHeFYY6m67Tq16rTR9XAyT=w6db5AyTXQ@mail.gmail.com, [2]/messages/by-id/CAJpy0uD6BXYTU2v1pQ3v4FFkGWJJaLJfsi32apNz0T1hPckP=w@mail.gmail.com, [3]/messages/by-id/CAHut+PuDFDiBSEqxejk+8EY1OjbmOYbDv1Gi5B3hWhZMYcot2w@mail.gmail.com,
[4]: /messages/by-id/CAJpy0uAi5RBMhBR+mYAOEcBWKJKckAKPvj2-a_XUnTsvR8jd3w@mail.gmail.com
Thanks Nisha for addressing comments in pg_dump.c and the sgml files.
In [6]/messages/by-id/CAHut+Pu3NRtUaFK3PD+SccqROUjtLkxbCmto7gRaXp2o_EQaLA@mail.gmail.com - I've not addressed the suggestion to change the \dRS+ output to
display conflict resolvers as a table footer. I will post that in the next
patch.
I've tried to implement all the changes in the nitpick patch in [6]/messages/by-id/CAHut+Pu3NRtUaFK3PD+SccqROUjtLkxbCmto7gRaXp2o_EQaLA@mail.gmail.com,
although I couldn't apply the patch directly as I had made too many changes
by then. If I've missed something, let me know.
We will target comments post Sep30 as well as the rest of the patches in
the next update.
[1]: /messages/by-id/CAJpy0uBS1_wKzmhP9uHeFYY6m67Tq16rTR9XAyT=w6db5AyTXQ@mail.gmail.com
/messages/by-id/CAJpy0uBS1_wKzmhP9uHeFYY6m67Tq16rTR9XAyT=w6db5AyTXQ@mail.gmail.com
[2]: /messages/by-id/CAJpy0uD6BXYTU2v1pQ3v4FFkGWJJaLJfsi32apNz0T1hPckP=w@mail.gmail.com
/messages/by-id/CAJpy0uD6BXYTU2v1pQ3v4FFkGWJJaLJfsi32apNz0T1hPckP=w@mail.gmail.com
[3]: /messages/by-id/CAHut+PuDFDiBSEqxejk+8EY1OjbmOYbDv1Gi5B3hWhZMYcot2w@mail.gmail.com
/messages/by-id/CAHut+PuDFDiBSEqxejk+8EY1OjbmOYbDv1Gi5B3hWhZMYcot2w@mail.gmail.com
[4]: /messages/by-id/CAJpy0uAi5RBMhBR+mYAOEcBWKJKckAKPvj2-a_XUnTsvR8jd3w@mail.gmail.com
/messages/by-id/CAJpy0uAi5RBMhBR+mYAOEcBWKJKckAKPvj2-a_XUnTsvR8jd3w@mail.gmail.com
[5]: /messages/by-id/CAJpy0uCqZjpz=MnfyrHQrzw0rDuuOTWGNDsyy_=2zpTJ6dx2Tg@mail.gmail.com
/messages/by-id/CAJpy0uCqZjpz=MnfyrHQrzw0rDuuOTWGNDsyy_=2zpTJ6dx2Tg@mail.gmail.com
[6]: /messages/by-id/CAHut+Pu3NRtUaFK3PD+SccqROUjtLkxbCmto7gRaXp2o_EQaLA@mail.gmail.com
/messages/by-id/CAHut+Pu3NRtUaFK3PD+SccqROUjtLkxbCmto7gRaXp2o_EQaLA@mail.gmail.com
regards,
Ajin Cherian
Fujitsu Australia
Attachments:
v15-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v15-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From a0ccc27f0423dbc4ea8c0888aed3920799265de5 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Mon, 7 Oct 2024 23:52:31 -0400
Subject: [PATCH v15] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers.
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for resetting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 95 +-----
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 62 ++++
doc/src/sgml/ref/create_subscription.sgml | 206 ++++++++++++
src/backend/commands/subscriptioncmds.c | 53 +++
src/backend/parser/gram.y | 49 ++-
src/backend/replication/logical/conflict.c | 426 ++++++++++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 116 ++++++-
src/bin/pg_dump/pg_dump.h | 8 +
src/bin/psql/describe.c | 15 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_subscription_conflict.h | 55 ++++
src/include/nodes/parsenodes.h | 7 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 ++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 251 ++++++++++-----
src/test/regress/sql/subscription.sql | 63 ++++
src/tools/pgindent/typedefs.list | 3 +
20 files changed, 1303 insertions(+), 174 deletions(-)
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 98a7ad0..86c9b0b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and Conflict Resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,86 +1582,19 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ There are various conflict scenarios, each identified as a <firstterm>conflict type</firstterm>.
+ Users can configure a <firstterm>conflict resolver</firstterm> for each
+ conflict type when creating a subscription. For more information, refer to
+ <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIPTION ... CONFLICT RESOLVER</command></link>.
+ </para>
+ <para>
+ When a conflict occurs the details about it are logged, and the conflict
+ statistics are recorded in the <link linkend="monitoring-pg-stat-subscription-stats">
+ <structname>pg_stat_subscription_stats</structname></link> view.
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the
+ log.
</para>
<para>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 331315f..d79db76 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d..e97edfe 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] )
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER ALL
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER FOR '<replaceable class="parameter">conflict_type</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -345,6 +348,65 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters the current conflict resolver for the specified conflict types.
+ Refer to <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIBER ... CONFLICT RESOLVER</command></link>
+ for details about different <literal>conflict_type</literal> and what
+ kind of <literal>conflict_resolver</literal> can be assigned to them.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-resolver">
+ <term><replaceable class="parameter">conflict_resolver</replaceable></term>
+ <listitem>
+ <para>
+ The conflict resolver to use for this conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-reset-conflict-resolver">
+ <term><literal>RESET CONFLICT RESOLVER</literal></term>
+ <listitem>
+ <para>
+ Conflict types can either be reset to their default resolvers all at once using <replaceable class="parameter">ALL</replaceable>, or a specific conflict type can be reset using <replaceable class="parameter">FOR 'conflict_type'</replaceable>.
+ For details on conflict types and their default resolvers, refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CREATE SUBSCRIBER ... CONFLICT RESOLVER</literal></link>.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-reset-all">
+ <term><literal>ALL</literal></term>
+ <listitem>
+ <para>
+ All conflict types will be reset to their respective default resolvers.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-reset-conflict-type">
+ <term><literal>FOR</literal> <replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The given <replaceable class="parameter">conflict_type</replaceable> will be reset to its default resolver.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8a3096e..91d5ffb 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -24,6 +24,7 @@ PostgreSQL documentation
CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceable>
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] ) ]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -97,6 +98,211 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The default behavior for each <replaceable class="parameter">conflict_type</replaceable> is listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists" xreflabel="insert_exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists" xreflabel="update_exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ <warning>
+ <para>
+ If the local table contains multiple <literal>NOT DEFERRABLE</literal> unique constraint
+ columns and the conflict resolution strategy for an INSERT or UPDATE favors applying the changes,
+ then a remote INSERT or UPDATE could trigger multiple <literal>update_exists</literal> conflicts.
+ Each conflict will be detected and resolved in sequence, potentially leading to the deletion of
+ multiple local rows.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing" xreflabel="update_missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs" xreflabel="update_origin_differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing" xreflabel="delete_missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs" xreflabel="delete_origin_differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ The behavior of each <replaceable class="parameter">conflict_resolver</replaceable>
+ is described below. Users can choose from the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ It is the default resolver for <literal>update_origin_differs</literal> and
+ <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <table id="sql-createsubscription-params-conflict-type-resolver-summary">
+ <title>Conflict type/resolver Summary</title>
+ <tgroup cols="3">
+ <thead>
+ <row><entry>Conflict type</entry> <entry>Default resolver</entry> <entry>Possible resolvers</entry></row>
+ </thead>
+ <tbody>
+ <row><entry>insert_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_missing</entry> <entry>skip</entry> <entry>apply_or_error, apply_or_skip, error, skip</entry></row>
+ <row><entry>update_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>delete_missing</entry> <entry>skip</entry> <entry>error, skip</entry></row>
+ <row><entry>delete_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createsubscription-params-with">
<term><literal>WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc63..54d8ad9 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -36,6 +36,7 @@
#include "executor/executor.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -583,6 +584,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ List *conflict_resolvers = NIL;
/*
* Parse and check options.
@@ -597,6 +599,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /* Parse and get conflict resolvers list. */
+ conflict_resolvers =
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -723,6 +729,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolvers(subid, conflict_resolvers);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1590,47 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Get the list of conflict types and resolvers and validate
+ * them.
+ */
+ conflict_resolvers = ParseAndGetSubConflictResolvers(
+ pstate,
+ stmt->resolvers,
+ false);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog.
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Create list of conflict resolvers and set them in the
+ * catalog.
+ */
+ conflict_resolvers = GetDefaultConflictResolvers();
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET:
+ {
+ /*
+ * Reset the conflict resolver for this conflict type to its
+ * default.
+ */
+ ResetSubConflictResolver(subid, stmt->conflict_type_name);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1882,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4aa8646..28591c9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -770,7 +771,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8813,6 +8814,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10747,14 +10753,15 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_resolver_definition opt_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
n->subname = $3;
n->conninfo = $5;
n->publication = $7;
- n->options = $8;
+ n->resolvers = $8;
+ n->options = $9;
$$ = (Node *) n;
}
;
@@ -10861,6 +10868,38 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET;
+ n->subname = $3;
+ n->conflict_type_name = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
;
/*****************************************************************************
@@ -17785,6 +17824,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18414,6 +18454,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff62..eb46e39 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -8,20 +8,28 @@
* src/backend/replication/logical/conflict.c
*
* This file contains the code for logging conflicts on the subscriber during
- * logical replication.
+ * logical replication and setting up conflict resolvers for a subscription.
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heapam.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -32,6 +40,50 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * First member represents default resolver for each conflict_type.
+ * The same defaults are used in pg_dump.c. If any default is changed here,
+ * ensure the corresponding value is updated in pg_dump's is_default_resolver
+ * function.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS + 1] = {
+ [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
+ [CT_UPDATE_MISSING] = {CR_SKIP, CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_ERROR, -1},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
+};
+
+StaticAssertDecl(lengthof(ConflictTypeResolverMap) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -489,3 +541,375 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
+}
+
+/*
+ * Validate the conflict type and return the corresponding ConflictType enum.
+ */
+ConflictType
+ValidateConflictType(const char *conflict_type)
+{
+ ConflictType type;
+ bool valid = false;
+
+ /* Check conflict type validity */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ return type;
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+
+ /* Validate conflict type */
+ type = ValidateConflictType(conflict_type);
+
+ /* Validate the conflict resolver name */
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+ }
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (int i = 0; i < CONFLICT_NUM_RESOLVERS; i++)
+ {
+ int candidate = ConflictTypeResolverMap[type][i];
+
+ if (candidate < 0)
+ /* No more possible resolvers for this conflict type */
+ break;
+
+ if (candidate == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+
+ return type;
+}
+
+/*
+ * Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
+ * SUBSCRIPTION commands.
+ *
+ * It reports an error if duplicate options are specified.
+ *
+ * Returns a list of conflict types along with their corresponding conflict
+ * resolvers. If 'add_defaults' is true, it appends default resolvers for any
+ * conflict types that have not been explicitly defined by the user.
+ */
+List *
+ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers, bool add_defaults)
+{
+ List *SeenTypes = NIL;
+ List *res = NIL;
+
+ /* Loop through the user provided resolvers */
+ foreach_ptr(DefElem, defel, stmtresolvers)
+ {
+ char *resolver;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ /* Check if the conflict type has already been seen */
+ if (list_member(SeenTypes, makeString(defel->defname)))
+ errorConflictingDefElem(defel, pstate);
+
+ /* Validate the conflict type and resolver */
+ resolver = defGetString(defel);
+ resolver = downcase_truncate_identifier(
+ resolver, strlen(resolver), false);
+ ValidateConflictTypeAndResolver(defel->defname,
+ resolver);
+
+ /* Add the conflict type to the list of seen types */
+ SeenTypes = lappend(SeenTypes, makeString(defel->defname));
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = defel->defname;
+ conftyperesolver->conflict_resolver_name = resolver;
+ res = lappend(res, conftyperesolver);
+ }
+
+ /* Once validation is complete, warn users if prerequisites are not met */
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+ /*
+ * If add_defaults is true, fill remaining conflict types with default
+ * resolvers.
+ */
+ if (add_defaults)
+ {
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ {
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ if (!list_member(SeenTypes, makeString((char *) ConflictTypeNames[i])))
+ {
+ ConflictResolver def_resolver = ConflictTypeResolverMap[i][0];
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = ConflictTypeNames[i];
+ conftyperesolver->conflict_resolver_name = ConflictResolverNames[def_resolver];;
+ res = lappend(res, conftyperesolver);
+ }
+ }
+ }
+
+ list_free_deep(SeenTypes);
+
+ return res;
+}
+
+/*
+ * Get the list of conflict types and their corresponding default resolvers.
+ */
+List *
+GetDefaultConflictResolvers()
+{
+ List *res = NIL;
+
+ for (ConflictType type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ ConflictTypeResolver *resolver = NULL;
+ ConflictResolver def_resolver = ConflictTypeResolverMap[type][0];
+
+ /* Allocate memory for each ConflictTypeResolver */
+ resolver = palloc(sizeof(ConflictTypeResolver));
+
+ resolver->conflict_type_name = ConflictTypeNames[type];
+ resolver->conflict_resolver_name = ConflictResolverNames[def_resolver];
+
+ /* Append to the response list */
+ res = lappend(res, resolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ /* Set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_conftype - 1]);
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type_name, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /*
+ * Update system catalog only if the new resolver is not same as the
+ * existing one.
+ */
+ if (pg_strcasecmp(cur_conflict_res,
+ conftyperesolver->conflict_resolver_name) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+ replaces[Anum_pg_subscription_conflict_confres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup,
+ RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict,
+ &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetSubConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver *conflictResolver = NULL;
+ List *conflictresolver_list = NIL;
+
+ /* Validate the conflict type and get the index */
+ idx = ValidateConflictType(conflict_type);
+ conflictResolver = palloc(sizeof(ConflictTypeResolver));
+ conflictResolver->conflict_type_name = conflict_type;
+
+ /* Get the default resolver for this conflict_type */
+ conflictResolver->conflict_resolver_name =
+ ConflictResolverNames[ConflictTypeResolverMap[idx][0]];
+
+ /* Create a list of conflict resolvers and update in catalog */
+ conflictresolver_list = lappend(conflictresolver_list, conflictResolver);
+ UpdateSubConflictResolvers(conflictresolver_list, subid);
+
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+
+ /* Iterate over the list of resolvers */
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] =
+ ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict,
+ SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] =
+ ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid to return all conflict resolvers for this
+ * subscription.
+ */
+ ScanKeyInit(&skey[0],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, 1, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1b47c38..572eb52 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -395,6 +395,8 @@ static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
static bool forcePartitionRootLoad(const TableInfo *tbinfo);
static void read_dump_filters(const char *filename, DumpOptions *dopt);
+static bool is_default_resolver(const char *confType, const char *confRes);
+static void destroyConcflictResolverList(SimplePtrList *list);
int
@@ -4830,7 +4832,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
+ PQExpBuffer confQuery;
PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4851,7 +4855,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5012,6 +5018,37 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT conftype, confres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u order by conftype;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+
+ /* Initialize pointers in the list to NULL */
+ subinfo[i].conflict_resolver = (SimplePtrList)
+ {
+ 0
+ };
+
+ /* Store conflict types and resolvers from the query result in subinfo */
+ for (j = 0; j < ntuples; j++)
+ {
+ /* Create the ConflictTypeResolver node */
+ ConflictTypeResolver *conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+
+ conftyperesolver->conflict_type = pg_strdup(PQgetvalue(confRes, j, 0));
+ conftyperesolver->resolver = pg_strdup(PQgetvalue(confRes, j, 1));
+
+ /* Append the node to subinfo's list */
+ simple_ptr_list_append(&subinfo[i].conflict_resolver, conftyperesolver);
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5257,6 +5294,35 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* Add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ bool first_resolver = true;
+ SimplePtrListCell *cell = NULL;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ for (cell = subinfo->conflict_resolver.head; cell; cell = cell->next)
+ {
+ conftyperesolver = (ConflictTypeResolver *) cell->ptr;
+
+ if (!is_default_resolver(conftyperesolver->conflict_type,
+ conftyperesolver->resolver))
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query, ", %s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ }
+ }
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
@@ -5315,6 +5381,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
NULL, subinfo->rolname,
subinfo->dobj.catId, 0, subinfo->dobj.dumpId);
+ /* Clean-up the conflict_resolver list */
+ destroyConcflictResolverList((SimplePtrList *) &subinfo->conflict_resolver);
+
destroyPQExpBuffer(publications);
free(pubnames);
@@ -19051,3 +19120,48 @@ read_dump_filters(const char *filename, DumpOptions *dopt)
filter_free(&fstate);
}
+
+/*
+ * is_default_resolver - checks if the given resolver is the default for the
+ * specified conflict type.
+ */
+static bool
+is_default_resolver(const char *confType, const char *confRes)
+{
+ /*
+ * The default resolvers for each conflict type are taken from the
+ * predefined mapping ConflictTypeDefaultResolvers[] in conflict.c.
+ *
+ * Only modify these defaults if the corresponding values in conflict.c
+ * are changed.
+ */
+
+ if (strcmp(confType, "insert_exists") == 0 ||
+ strcmp(confType, "update_exists") == 0)
+ return strcmp(confRes, "error") == 0;
+ else if (strcmp(confType, "update_missing") == 0 ||
+ strcmp(confType, "delete_missing") == 0)
+ return strcmp(confRes, "skip") == 0;
+ else if (strcmp(confType, "update_origin_differs") == 0 ||
+ strcmp(confType, "delete_origin_differs") == 0)
+ return strcmp(confRes, "apply_remote") == 0;
+
+ return false;
+}
+
+/*
+ * destroyConflictResolverList - frees up the SimplePtrList containing
+ * cells pointing to struct ConflictTypeResolver nodes.
+ */
+static void
+destroyConcflictResolverList(SimplePtrList *conflictlist)
+{
+ SimplePtrListCell *cell = NULL;
+
+ /* Free the list items */
+ for (cell = conflictlist->head; cell; cell = cell->next)
+ pfree((ConflictTypeResolver *) cell->ptr);
+
+ /* Destroy the pointer list */
+ simple_ptr_list_destroy(conflictlist);
+}
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 9f907ed..bb16e8b 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,13 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
@@ -673,6 +680,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ SimplePtrList conflict_resolver;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 6a36c91..8599f22 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6547,7 +6547,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6627,12 +6627,21 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* Add conflict resolvers information from pg_subscription_conflict */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", (SELECT string_agg(conftype || ' = ' || confres, ', ' \n"
+ " ORDER BY conftype) \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " WHERE c.confsubid = s.oid) AS \"%s\"\n",
+ gettext_noop("Conflict Resolvers"));
}
/* Only display subscriptions in current database. */
appendPQExpBufferStr(&buf,
- "FROM pg_catalog.pg_subscription\n"
- "WHERE subdbid = (SELECT oid\n"
+ "FROM pg_catalog.pg_subscription s\n"
+ "WHERE s.subdbid = (SELECT oid\n"
" FROM pg_catalog.pg_database\n"
" WHERE datname = pg_catalog.current_database())");
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a..f2611c1 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1da..959e1d9 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000..15f02a1
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict *Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_confsubid_conftype_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, conftype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_confsubid_conftype_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1c314cd..cd934f7 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4216,6 +4216,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4228,6 +4229,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4238,6 +4242,9 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict type names and resolver
+ * names */
+ char *conflict_type_name; /* The conflict type name to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 899d64a..e62b0c6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -376,6 +376,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677..c5865a1 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -1,6 +1,6 @@
/*-------------------------------------------------------------------------
* conflict.h
- * Exports for conflicts logging.
+ * Exports for conflicts logging and resolvers configuration.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "parser/parse_node.h"
#include "utils/timestamp.h"
/*
@@ -50,6 +51,41 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it cannot be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it cannot be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+#define CONFLICT_NUM_RESOLVERS (CR_ERROR + 1)
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type_name;
+ const char *conflict_resolver_name;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +98,16 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolvers(Oid subId, List *resolvers);
+extern void RemoveSubConflictResolvers(Oid confid);
+extern List *ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers,
+ bool add_defaults);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType ValidateConflictType(const char *conflict_type);
+extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern List *GetDefaultConflictResolvers(void);
+extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb89..42bf2cc 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..9791141 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,109 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+ERROR: foo is not a valid conflict type
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+ERROR: conflicting or redundant options
+LINE 1: ... CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exi...
+ ^
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = keep_local, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+ERROR: conflicting or redundant options
+LINE 1: ...ONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exi...
+ ^
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = error, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = keep_local, update_missing = skip, update_origin_differs = error
+(1 row)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = error, delete_origin_differs = keep_local, insert_exists = error, update_exists = keep_local, update_missing = skip, update_origin_differs = error
+(1 row)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +508,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..3f08869 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,69 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+
+\dRs+
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+
+\dRs+
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+
+\dRs+
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a65e1c0..bf1ea95 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -468,7 +468,9 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
ConflictType
+ConflictTypeResolver
ConnCacheEntry
ConnCacheKey
ConnParams
@@ -863,6 +865,7 @@ FormData_pg_statistic
FormData_pg_statistic_ext
FormData_pg_statistic_ext_data
FormData_pg_subscription
+FormData_pg_subscription_conflict
FormData_pg_subscription_rel
FormData_pg_tablespace
FormData_pg_transform
--
1.8.3.1
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
Here are some review comments for v14-0001.
~~~
7.
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable>
RESET CONFLICT RESOLVER FOR (<replaceable
class="parameter">conflict_type</replaceable>)I can see that this matches the implementation, but I was wondering
why don't you permit resetting multiple conflict_types at the same
time. e.g. what if I want to reset some but not ALL?
Thank you for your input.
The RESET command was not part of the initial design, and our current
implementation for resetting ALL or a specific 'conflict_type'
effectively serves its purpose.
Allowing the option to reset two or more conflict types in one command
may complicate the implementation. However, if others also feel the
same, we can implement it. Let's wait for others' feedback.
--
Thanks,
Nisha
On Mon, Sep 30, 2024 at 11:59 AM shveta malik <shveta.malik@gmail.com> wrote:
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
...
13. General - ordering of conflict_type.
nit - Instead of just some apparent random order, let's put each
insert/update/delete conflict type in alphabetical order, so at least
users can find them where they would expect to find them.This ordering was decided while implementing the 'conflict-detection
and logging' patch and thus perhaps should be maintained as same. The
ordering is insert, update and delete (different variants of these).
Please see a comment on it in [1] (comment #2).[1]:/messages/by-id/TYAPR01MB569224262F44875973FAF344F5B22@TYAPR01MB5692.jpnprd01.prod.outlook.com
+1 for order insert/update/delete.
My issue was only about the order *within* each of those variants.
e.g. I think it should be alphabetical:CURRENT
insert_exists
update_origin_differs
update_exists
update_missing
delete_origin_differs
delete_missingSUGGESTED
insert_exists
update_exists
update_missing
update_origin_differs
delete_missing
delete_origin_differsOkay, got it now. I have no strong opinion here. I am okay with both.
But since it was originally added by other thread, so it will be good
to know the respective author's opinion as well.
v15 has the above "SUGGESTED" order of conflict_type. We can update it
if the original thread's author or others have different preferences.
Thanks,
Nisha
On Tue, Oct 1, 2024 at 9:48 AM shveta malik <shveta.malik@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:55 PM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 4:29 PM shveta malik <shveta.malik@gmail.com> wrote:
On Mon, Sep 30, 2024 at 11:04 AM Peter Smith <smithpb2250@gmail.com> wrote:
On Mon, Sep 30, 2024 at 2:27 PM shveta malik <shveta.malik@gmail.com> wrote:
On Fri, Sep 27, 2024 at 1:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
~~~
14.
99. General - ordering of conflict_resolvernit - ditto. Let's name these in alphabetical order. IMO it makes more
sense than the current random ordering.I feel ordering of resolvers should be same as that of conflict
types, i.e. resolvers of insert variants first, then update variants,
then delete variants. But would like to know what others think on
this.Resolvers in v14 were documented in this random order:
error
skip
apply_remote
keep_local
apply_or_skip
apply_or_errorYes, these should be changed.
Some of these are resolvers for different conflicts. How can you order
these as "resolvers for insert" followed by "resolvers for update"
followed by "resolvers for delete" without it all still appearing in
random order?I was thinking of ordering them like this:
apply_remote: applicable to insert_exists, update_exists,
update_origin_differ, delete_origin_differ
keep_local: applicable to insert_exists,
update_exists, update_origin_differ, delete_origin_differ
apply_or_skip: applicable to update_missing
apply_or_error : applicable to update_missing
skip: applicable to update_missing and
delete_missing
error: applicable to all.i.e. in order of how they are applicable to conflict_types starting
from insert_exists till delete_origin_differ (i.e. reading
ConflictTypeResolverMap, from left to right and then top to bottom).
Except I have kept 'error' at the end instead of keeping it after
'keep_local' as the former makes more sense there.This proves my point because, without your complicated explanation to
accompany it, the final order (below) just looks random to me:
apply_remote
keep_local
apply_or_skip
apply_or_error
skip
errorUnless there is some compelling reason to do it differently, I still
prefer A-Z (the KISS principle).The "applicable to conflict_types" against each resolver (which will
be mentioned in doc too) is a pretty good reason in itself to keep the
resolvers in the suggested order. To me, it seems more logical than
placing 'apply_or_error' which only applies to the 'update_missing'
conflict_type at the top, while 'error,' which applies to all
conflict_types, placed in the middle. But I understand that
preferences may vary, so I'll leave this to the discretion of others.
In v15, I maintained the original order of conflict_resolver, which to
me seems reasonable from a user perspective:
error
skip
apply_remote
keep_local
apply_or_error
apply_or_skip
I will hold this order until we receive feedback from others, and we
can finalize the new order if necessary.
Thanks,
Nisha
On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
I have not started reviewing v15 yet, but here are few comments for
v14-patch003:
1)
In apply_handle_update_internal(), I see that
FindReplTupleInLocalRel() used to lock the row to be updated in
exclusive mode, but now since we are avoiding this call in recursion
and thus the second call onwards, the tuple to be updated or deleted
will not be locked in exclusive mode. You are perhaps locking it
(conflictslot passsed as localslot in recursion) somewhere in
FindConflictTuple in shared mode. Since we are going to delete/update
this tuple, shouldn't it be locked in exclusive mode?
2)
Also, for multiple-key violations case, it would be good to verify the
behavior that when say last update fails for some reason, all the
deleted rows are reverted back? It seems so, but please test once by
forcing the last operation to fail.
thanks
Shveta
On Wed, Oct 9, 2024 at 8:58 AM shveta malik <shveta.malik@gmail.com> wrote:
On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond <nisha.moond412@gmail.com> wrote:
Please find few comments on v14-patch004:
patch004:
1)
GetConflictResolver currently errors out when the resolver is
last_update_wins and track_commit_timestamp is disabled. It means
every conflict resolution with this resolver will keep on erroring
out. I am not sure if we should emit ERROR here. We do emit ERROR when
someone tries to configure last_update_wins but track_commit_timestamp
is disabled. I think that should suffice. The one in
GetConflictResolver can be converted to WARNING max.
What could be the side-effect if we do not emit error here? In such a
case, the local timestamp will be 0 and remote change will always win.
Is that right? If so, then if needed, we can emit a warning saying
something like: 'track_commit_timestamp is disabled and thus remote
change is applied always.'
Thoughts?
2)
execReplication.c:
There are some optimizations in this file (moving duplicate code to
has_conflicting_tuple), I think these optimizations are applicable
even to patch003 (or patch002 as well?) and thus can be moved there.
Please review once.
thanks
Shveta
Here is the v16 patch-set.
- Rebased the 002 to 004 patches on top of the v15-001 patch.
- Addressed patch-002 review comments in [1]/messages/by-id/CAJpy0uA4jU31b6NJxJV0Bt2fC2o4d99RP380SHhoZHUc0MydoQ@mail.gmail.com and [2]/messages/by-id/CAJpy0uA3d0VC4+KNiLx8zVf7465iMNHse5rweGCNQpwAS8cRmA@mail.gmail.com and some offlist
comments provided by Shveta.
[1]: /messages/by-id/CAJpy0uA4jU31b6NJxJV0Bt2fC2o4d99RP380SHhoZHUc0MydoQ@mail.gmail.com
[2]: /messages/by-id/CAJpy0uA3d0VC4+KNiLx8zVf7465iMNHse5rweGCNQpwAS8cRmA@mail.gmail.com
Thanks,
Nisha
Attachments:
v16-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v16-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From b6b02ae07051d8a57fad5cfa0d599ed2a228e4dc Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Mon, 7 Oct 2024 23:52:31 -0400
Subject: [PATCH v16 1/4] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers.
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for resetting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 95 +---
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 62 +++
doc/src/sgml/ref/create_subscription.sgml | 206 +++++++++
src/backend/commands/subscriptioncmds.c | 53 +++
src/backend/parser/gram.y | 49 +-
src/backend/replication/logical/conflict.c | 426 +++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 116 ++++-
src/bin/pg_dump/pg_dump.h | 8 +
src/bin/psql/describe.c | 15 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
.../catalog/pg_subscription_conflict.h | 55 +++
src/include/nodes/parsenodes.h | 7 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 +-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 251 +++++++----
src/test/regress/sql/subscription.sql | 63 +++
src/tools/pgindent/typedefs.list | 3 +
20 files changed, 1303 insertions(+), 174 deletions(-)
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 98a7ad0c27..86c9b0b671 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and Conflict Resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,86 +1582,19 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ There are various conflict scenarios, each identified as a <firstterm>conflict type</firstterm>.
+ Users can configure a <firstterm>conflict resolver</firstterm> for each
+ conflict type when creating a subscription. For more information, refer to
+ <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIPTION ... CONFLICT RESOLVER</command></link>.
+ </para>
+ <para>
+ When a conflict occurs the details about it are logged, and the conflict
+ statistics are recorded in the <link linkend="monitoring-pg-stat-subscription-stats">
+ <structname>pg_stat_subscription_stats</structname></link> view.
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the
+ log.
</para>
<para>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 331315f8d3..d79db76e16 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d007..e97edfecba 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] )
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER ALL
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER FOR '<replaceable class="parameter">conflict_type</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -345,6 +348,65 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters the current conflict resolver for the specified conflict types.
+ Refer to <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIBER ... CONFLICT RESOLVER</command></link>
+ for details about different <literal>conflict_type</literal> and what
+ kind of <literal>conflict_resolver</literal> can be assigned to them.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-resolver">
+ <term><replaceable class="parameter">conflict_resolver</replaceable></term>
+ <listitem>
+ <para>
+ The conflict resolver to use for this conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-reset-conflict-resolver">
+ <term><literal>RESET CONFLICT RESOLVER</literal></term>
+ <listitem>
+ <para>
+ Conflict types can either be reset to their default resolvers all at once using <replaceable class="parameter">ALL</replaceable>, or a specific conflict type can be reset using <replaceable class="parameter">FOR 'conflict_type'</replaceable>.
+ For details on conflict types and their default resolvers, refer to section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CREATE SUBSCRIBER ... CONFLICT RESOLVER</literal></link>.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-reset-all">
+ <term><literal>ALL</literal></term>
+ <listitem>
+ <para>
+ All conflict types will be reset to their respective default resolvers.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-reset-conflict-type">
+ <term><literal>FOR</literal> <replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The given <replaceable class="parameter">conflict_type</replaceable> will be reset to its default resolver.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8a3096e62b..91d5ffbab4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -24,6 +24,7 @@ PostgreSQL documentation
CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceable>
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] ) ]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -97,6 +98,211 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies conflict resolvers for different conflict_types.
+ </para>
+
+ <para>
+ The default behavior for each <replaceable class="parameter">conflict_type</replaceable> is listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists" xreflabel="insert_exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists" xreflabel="update_exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint. To log the origin and commit
+ timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber. In this case, an error will be
+ raised until the conflict is resolved manually or the resolver is configured to a
+ non-default value that can automatically resolve the conflict.
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ <warning>
+ <para>
+ If the local table contains multiple <literal>NOT DEFERRABLE</literal> unique constraint
+ columns and the conflict resolution strategy for an INSERT or UPDATE favors applying the changes,
+ then a remote INSERT or UPDATE could trigger multiple <literal>update_exists</literal> conflicts.
+ Each conflict will be detected and resolved in sequence, potentially leading to the deletion of
+ multiple local rows.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing" xreflabel="update_missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The update will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs" xreflabel="update_origin_differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the update is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing" xreflabel="delete_missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The delete will simply be skipped in this scenario.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs" xreflabel="delete_origin_differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin. Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber. Currently, the delete is always applied
+ regardless of the origin of the local row.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ The behavior of each <replaceable class="parameter">conflict_resolver</replaceable>
+ is described below. Users can choose from the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication. It can be used for
+ any conflict type.
+ It is the default resolver for <literal>insert_exists</literal> and
+ <literal>update_exists</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continue replication
+ with the next change.
+ It can be used for <literal>update_missing</literal> and
+ <literal>delete_missing</literal> and is the default resolver for these types.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change. It can be used for
+ <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ It is the default resolver for <literal>update_origin_differs</literal> and
+ <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ With this resolver, the remote change is not applied and the local tuple is maintained.
+ It can be used for <literal>insert_exists</literal>, <literal>update_exists</literal>,
+ <literal>update_origin_differs</literal> and <literal>delete_origin_differs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <table id="sql-createsubscription-params-conflict-type-resolver-summary">
+ <title>Conflict type/resolver Summary</title>
+ <tgroup cols="3">
+ <thead>
+ <row><entry>Conflict type</entry> <entry>Default resolver</entry> <entry>Possible resolvers</entry></row>
+ </thead>
+ <tbody>
+ <row><entry>insert_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_missing</entry> <entry>skip</entry> <entry>apply_or_error, apply_or_skip, error, skip</entry></row>
+ <row><entry>update_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>delete_missing</entry> <entry>skip</entry> <entry>error, skip</entry></row>
+ <row><entry>delete_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createsubscription-params-with">
<term><literal>WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc636b8..54d8ad9554 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -36,6 +36,7 @@
#include "executor/executor.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -583,6 +584,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ List *conflict_resolvers = NIL;
/*
* Parse and check options.
@@ -597,6 +599,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /* Parse and get conflict resolvers list. */
+ conflict_resolvers =
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -723,6 +729,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolvers(subid, conflict_resolvers);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1590,47 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Get the list of conflict types and resolvers and validate
+ * them.
+ */
+ conflict_resolvers = ParseAndGetSubConflictResolvers(
+ pstate,
+ stmt->resolvers,
+ false);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog.
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Create list of conflict resolvers and set them in the
+ * catalog.
+ */
+ conflict_resolvers = GetDefaultConflictResolvers();
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET:
+ {
+ /*
+ * Reset the conflict resolver for this conflict type to its
+ * default.
+ */
+ ResetSubConflictResolver(subid, stmt->conflict_type_name);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1882,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4aa8646af7..28591c90c9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -770,7 +771,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8813,6 +8814,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10747,14 +10753,15 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_resolver_definition opt_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
n->subname = $3;
n->conninfo = $5;
n->publication = $7;
- n->options = $8;
+ n->resolvers = $8;
+ n->options = $9;
$$ = (Node *) n;
}
;
@@ -10861,6 +10868,38 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET;
+ n->subname = $3;
+ n->conflict_type_name = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
;
/*****************************************************************************
@@ -17785,6 +17824,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18414,6 +18454,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff626bd..eb46e39013 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -8,20 +8,28 @@
* src/backend/replication/logical/conflict.c
*
* This file contains the code for logging conflicts on the subscriber during
- * logical replication.
+ * logical replication and setting up conflict resolvers for a subscription.
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heapam.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
static const char *const ConflictTypeNames[] = {
[CT_INSERT_EXISTS] = "insert_exists",
@@ -32,6 +40,50 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
+
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * First member represents default resolver for each conflict_type.
+ * The same defaults are used in pg_dump.c. If any default is changed here,
+ * ensure the corresponding value is updated in pg_dump's is_default_resolver
+ * function.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS + 1] = {
+ [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
+ [CT_UPDATE_MISSING] = {CR_SKIP, CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_ERROR, -1},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
+};
+
+StaticAssertDecl(lengthof(ConflictTypeResolverMap) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -489,3 +541,375 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
+}
+
+/*
+ * Validate the conflict type and return the corresponding ConflictType enum.
+ */
+ConflictType
+ValidateConflictType(const char *conflict_type)
+{
+ ConflictType type;
+ bool valid = false;
+
+ /* Check conflict type validity */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ return type;
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+
+ /* Validate conflict type */
+ type = ValidateConflictType(conflict_type);
+
+ /* Validate the conflict resolver name */
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+ }
+
+ /* Reset */
+ valid = false;
+
+ /* Check if conflict resolver is a valid one for the given conflict type */
+ for (int i = 0; i < CONFLICT_NUM_RESOLVERS; i++)
+ {
+ int candidate = ConflictTypeResolverMap[type][i];
+
+ if (candidate < 0)
+ /* No more possible resolvers for this conflict type */
+ break;
+
+ if (candidate == resolver)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+
+ return type;
+}
+
+/*
+ * Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
+ * SUBSCRIPTION commands.
+ *
+ * It reports an error if duplicate options are specified.
+ *
+ * Returns a list of conflict types along with their corresponding conflict
+ * resolvers. If 'add_defaults' is true, it appends default resolvers for any
+ * conflict types that have not been explicitly defined by the user.
+ */
+List *
+ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers, bool add_defaults)
+{
+ List *SeenTypes = NIL;
+ List *res = NIL;
+
+ /* Loop through the user provided resolvers */
+ foreach_ptr(DefElem, defel, stmtresolvers)
+ {
+ char *resolver;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ /* Check if the conflict type has already been seen */
+ if (list_member(SeenTypes, makeString(defel->defname)))
+ errorConflictingDefElem(defel, pstate);
+
+ /* Validate the conflict type and resolver */
+ resolver = defGetString(defel);
+ resolver = downcase_truncate_identifier(
+ resolver, strlen(resolver), false);
+ ValidateConflictTypeAndResolver(defel->defname,
+ resolver);
+
+ /* Add the conflict type to the list of seen types */
+ SeenTypes = lappend(SeenTypes, makeString(defel->defname));
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = defel->defname;
+ conftyperesolver->conflict_resolver_name = resolver;
+ res = lappend(res, conftyperesolver);
+ }
+
+ /* Once validation is complete, warn users if prerequisites are not met */
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+ /*
+ * If add_defaults is true, fill remaining conflict types with default
+ * resolvers.
+ */
+ if (add_defaults)
+ {
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ {
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ if (!list_member(SeenTypes, makeString((char *) ConflictTypeNames[i])))
+ {
+ ConflictResolver def_resolver = ConflictTypeResolverMap[i][0];
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = ConflictTypeNames[i];
+ conftyperesolver->conflict_resolver_name = ConflictResolverNames[def_resolver];;
+ res = lappend(res, conftyperesolver);
+ }
+ }
+ }
+
+ list_free_deep(SeenTypes);
+
+ return res;
+}
+
+/*
+ * Get the list of conflict types and their corresponding default resolvers.
+ */
+List *
+GetDefaultConflictResolvers()
+{
+ List *res = NIL;
+
+ for (ConflictType type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ ConflictTypeResolver *resolver = NULL;
+ ConflictResolver def_resolver = ConflictTypeResolverMap[type][0];
+
+ /* Allocate memory for each ConflictTypeResolver */
+ resolver = palloc(sizeof(ConflictTypeResolver));
+
+ resolver->conflict_type_name = ConflictTypeNames[type];
+ resolver->conflict_resolver_name = ConflictResolverNames[def_resolver];
+
+ /* Append to the response list */
+ res = lappend(res, resolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ /* Set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_conftype - 1]);
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type_name, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /*
+ * Update system catalog only if the new resolver is not same as the
+ * existing one.
+ */
+ if (pg_strcasecmp(cur_conflict_res,
+ conftyperesolver->conflict_resolver_name) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+ replaces[Anum_pg_subscription_conflict_confres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup,
+ RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict,
+ &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetSubConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver *conflictResolver = NULL;
+ List *conflictresolver_list = NIL;
+
+ /* Validate the conflict type and get the index */
+ idx = ValidateConflictType(conflict_type);
+ conflictResolver = palloc(sizeof(ConflictTypeResolver));
+ conflictResolver->conflict_type_name = conflict_type;
+
+ /* Get the default resolver for this conflict_type */
+ conflictResolver->conflict_resolver_name =
+ ConflictResolverNames[ConflictTypeResolverMap[idx][0]];
+
+ /* Create a list of conflict resolvers and update in catalog */
+ conflictresolver_list = lappend(conflictresolver_list, conflictResolver);
+ UpdateSubConflictResolvers(conflictresolver_list, subid);
+
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+
+ /* Iterate over the list of resolvers */
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] =
+ ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict,
+ SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] =
+ ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid to return all conflict resolvers for this
+ * subscription.
+ */
+ ScanKeyInit(&skey[0],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, 1, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1b47c388ce..572eb52a5e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -395,6 +395,8 @@ static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
static bool forcePartitionRootLoad(const TableInfo *tbinfo);
static void read_dump_filters(const char *filename, DumpOptions *dopt);
+static bool is_default_resolver(const char *confType, const char *confRes);
+static void destroyConcflictResolverList(SimplePtrList *list);
int
@@ -4830,7 +4832,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
+ PQExpBuffer confQuery;
PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4851,7 +4855,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5012,6 +5018,37 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT conftype, confres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u order by conftype;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ ntuples = PQntuples(confRes);
+
+ /* Initialize pointers in the list to NULL */
+ subinfo[i].conflict_resolver = (SimplePtrList)
+ {
+ 0
+ };
+
+ /* Store conflict types and resolvers from the query result in subinfo */
+ for (j = 0; j < ntuples; j++)
+ {
+ /* Create the ConflictTypeResolver node */
+ ConflictTypeResolver *conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+
+ conftyperesolver->conflict_type = pg_strdup(PQgetvalue(confRes, j, 0));
+ conftyperesolver->resolver = pg_strdup(PQgetvalue(confRes, j, 1));
+
+ /* Append the node to subinfo's list */
+ simple_ptr_list_append(&subinfo[i].conflict_resolver, conftyperesolver);
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5257,6 +5294,35 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
if (pg_strcasecmp(subinfo->suborigin, LOGICALREP_ORIGIN_ANY) != 0)
appendPQExpBuffer(query, ", origin = %s", subinfo->suborigin);
+ /* Add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ bool first_resolver = true;
+ SimplePtrListCell *cell = NULL;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ for (cell = subinfo->conflict_resolver.head; cell; cell = cell->next)
+ {
+ conftyperesolver = (ConflictTypeResolver *) cell->ptr;
+
+ if (!is_default_resolver(conftyperesolver->conflict_type,
+ conftyperesolver->resolver))
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query, ", %s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ }
+ }
+ }
+
appendPQExpBufferStr(query, ");\n");
/*
@@ -5315,6 +5381,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
NULL, subinfo->rolname,
subinfo->dobj.catId, 0, subinfo->dobj.dumpId);
+ /* Clean-up the conflict_resolver list */
+ destroyConcflictResolverList((SimplePtrList *) &subinfo->conflict_resolver);
+
destroyPQExpBuffer(publications);
free(pubnames);
@@ -19051,3 +19120,48 @@ read_dump_filters(const char *filename, DumpOptions *dopt)
filter_free(&fstate);
}
+
+/*
+ * is_default_resolver - checks if the given resolver is the default for the
+ * specified conflict type.
+ */
+static bool
+is_default_resolver(const char *confType, const char *confRes)
+{
+ /*
+ * The default resolvers for each conflict type are taken from the
+ * predefined mapping ConflictTypeDefaultResolvers[] in conflict.c.
+ *
+ * Only modify these defaults if the corresponding values in conflict.c
+ * are changed.
+ */
+
+ if (strcmp(confType, "insert_exists") == 0 ||
+ strcmp(confType, "update_exists") == 0)
+ return strcmp(confRes, "error") == 0;
+ else if (strcmp(confType, "update_missing") == 0 ||
+ strcmp(confType, "delete_missing") == 0)
+ return strcmp(confRes, "skip") == 0;
+ else if (strcmp(confType, "update_origin_differs") == 0 ||
+ strcmp(confType, "delete_origin_differs") == 0)
+ return strcmp(confRes, "apply_remote") == 0;
+
+ return false;
+}
+
+/*
+ * destroyConflictResolverList - frees up the SimplePtrList containing
+ * cells pointing to struct ConflictTypeResolver nodes.
+ */
+static void
+destroyConcflictResolverList(SimplePtrList *conflictlist)
+{
+ SimplePtrListCell *cell = NULL;
+
+ /* Free the list items */
+ for (cell = conflictlist->head; cell; cell = cell->next)
+ pfree((ConflictTypeResolver *) cell->ptr);
+
+ /* Destroy the pointer list */
+ simple_ptr_list_destroy(conflictlist);
+}
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 9f907ed5ad..bb16e8b535 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,13 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
@@ -673,6 +680,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ SimplePtrList conflict_resolver;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 6a36c91083..8599f22044 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6547,7 +6547,7 @@ describeSubscriptions(const char *pattern, bool verbose)
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6627,12 +6627,21 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
+ /* Add conflict resolvers information from pg_subscription_conflict */
+ if (pset.sversion >= 180000)
+ appendPQExpBuffer(&buf,
+ ", (SELECT string_agg(conftype || ' = ' || confres, ', ' \n"
+ " ORDER BY conftype) \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " WHERE c.confsubid = s.oid) AS \"%s\"\n",
+ gettext_noop("Conflict Resolvers"));
}
/* Only display subscriptions in current database. */
appendPQExpBufferStr(&buf,
- "FROM pg_catalog.pg_subscription\n"
- "WHERE subdbid = (SELECT oid\n"
+ "FROM pg_catalog.pg_subscription s\n"
+ "WHERE s.subdbid = (SELECT oid\n"
" FROM pg_catalog.pg_database\n"
" WHERE datname = pg_catalog.current_database())");
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a6e3..f2611c1424 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1daba5..959e1d9ded 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000000..15f02a1d87
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict *Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_confsubid_conftype_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, conftype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_confsubid_conftype_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 1c314cd907..cd934f7475 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4216,6 +4216,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4228,6 +4229,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4238,6 +4242,9 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict type names and resolver
+ * names */
+ char *conflict_type_name; /* The conflict type name to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 899d64ad55..e62b0c6945 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -376,6 +376,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677ff5..c5865a1d58 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -1,6 +1,6 @@
/*-------------------------------------------------------------------------
* conflict.h
- * Exports for conflicts logging.
+ * Exports for conflicts logging and resolvers configuration.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "parser/parse_node.h"
#include "utils/timestamp.h"
/*
@@ -50,6 +51,41 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it cannot be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it cannot be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+#define CONFLICT_NUM_RESOLVERS (CR_ERROR + 1)
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type_name;
+ const char *conflict_resolver_name;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +98,16 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolvers(Oid subId, List *resolvers);
+extern void RemoveSubConflictResolvers(Oid confid);
+extern List *ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers,
+ bool add_defaults);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType ValidateConflictType(const char *conflict_type);
+extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern List *GetDefaultConflictResolvers(void);
+extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..42bf2cce92 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1685..97911415a4 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -116,18 +116,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub3;
@@ -145,10 +145,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -157,10 +157,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname');
ALTER SUBSCRIPTION regress_testsub SET (password_required = false);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
@@ -176,10 +176,10 @@ ERROR: unrecognized subscription parameter: "create_slot"
-- ok
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- ok - with lsn = NONE
@@ -188,10 +188,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0');
ERROR: invalid WAL location (LSN): 0/0
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
BEGIN;
@@ -223,10 +223,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- rename back to keep the rest simple
@@ -255,19 +255,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -279,27 +279,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- fail - publication already exists
@@ -314,10 +314,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false);
ERROR: publication "testpub1" is already in subscription "regress_testsub"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- fail - publication used more than once
@@ -332,10 +332,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub"
-- ok - delete publications
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
DROP SUBSCRIPTION regress_testsub;
@@ -371,19 +371,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -393,10 +393,109 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+ERROR: foo is not a valid conflict type
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+ERROR: conflicting or redundant options
+LINE 1: ... CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exi...
+ ^
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = keep_local, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+ERROR: conflicting or redundant options
+LINE 1: ...ONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exi...
+ ^
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
+(1 row)
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = error, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = keep_local, update_missing = skip, update_origin_differs = error
+(1 row)
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = error, delete_origin_differs = keep_local, insert_exists = error, update_exists = keep_local, update_missing = skip, update_origin_differs = error
+(1 row)
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -409,18 +508,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
WARNING: subscription was created, but is not connected
HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
- regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = apply_remote, insert_exists = error, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7037..3f08869bce 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,69 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+
+\dRs+
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+
+\dRs+
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+
+\dRs+
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a65e1c07c5..bf1ea95f18 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -468,7 +468,9 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
ConflictType
+ConflictTypeResolver
ConnCacheEntry
ConnCacheKey
ConnParams
@@ -863,6 +865,7 @@ FormData_pg_statistic
FormData_pg_statistic_ext
FormData_pg_statistic_ext_data
FormData_pg_subscription
+FormData_pg_subscription_conflict
FormData_pg_subscription_rel
FormData_pg_tablespace
FormData_pg_transform
--
2.34.1
v16-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v16-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 0f8ad507771d3de6a0c3d9e054a661e2b83a401a Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Tue, 8 Oct 2024 17:53:56 +0530
Subject: [PATCH v16 2/4] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 97 +-
src/backend/replication/logical/conflict.c | 230 ++++-
src/backend/replication/logical/worker.c | 374 ++++++--
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 12 +-
src/test/subscription/meson.build | 1 +
.../subscription/t/034_conflict_resolver.pl | 877 ++++++++++++++++++
7 files changed, 1442 insertions(+), 154 deletions(-)
create mode 100644 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 54025c9f15..27d6e98426 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,13 +550,75 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
+/*
+ * Check the unique indexes for conflicts. Return true on finding the
+ * first conflict itself.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(Oid subid, ConflictType type,
+ ResultRelInfo *resultRelInfo, EState *estate,
+ TupleTableSlot *slot, TupleTableSlot **conflictslot)
+{
+ ConflictResolver resolver;
+ bool apply_remote = false;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* ASSERT if called for any conflict type other than insert_exists */
+ Assert(type == CT_INSERT_EXISTS);
+
+ /*
+ * Get the configured resolver and determine if remote changes should be
+ * applied.
+ */
+ resolver = GetConflictResolver(subid, type, rel, NULL, &apply_remote);
+
+ /*
+ * Proceed to find conflict if the resolver is set to a non-default value;
+ * if the resolver is 'ERROR' (default), the caller will handle it.
+ */
+ if (resolver == CR_ERROR)
+ return false;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /* Return to caller for resolutions if any conflict is found */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
@@ -565,7 +627,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -601,6 +664,19 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Check for conflict and return to caller for resolution, if found.
+ *
+ * XXX In case there are no conflicts, a non-default 'insert_exists'
+ * resolver adds overhead by performing an extra scan here. However,
+ * this approach avoids the extra work needed to rollback/delete the
+ * inserted tuple if a conflict is detected after insertion with a
+ * non-default resolution set.
+ */
+ if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, resultRelInfo,
+ estate, slot, &(*conflictslot)))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
@@ -615,13 +691,14 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/*
* Checks the conflict indexes to fetch the conflicting local tuple
- * and reports the conflict. We perform this check here, instead of
- * performing an additional index scan before the actual insertion and
- * reporting the conflict if any conflicting tuples are found. This is
- * to avoid the overhead of executing the extra scan for each INSERT
- * operation, even when no conflict arises, which could introduce
- * significant overhead to replication, particularly in cases where
- * conflicts are rare.
+ * and reports the conflict. We perform this check here again to -
+ *
+ * a) optimize the default case where the resolution for
+ * 'insert_exists' is set to 'ERROR' by skipping the scan when there
+ * is no conflict.
+ *
+ * b) catch and report any conflict that might have been missed during
+ * the pre-insertion scan in has_conflicting_tuple().
*
* XXX OTOH, this could lead to clean-up effort for dead tuples added
* in heap and index in case of conflicts. But as conflicts shouldn't
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index eb46e39013..8309a70883 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -24,9 +24,9 @@
#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "replication/worker_internal.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
-#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -88,12 +88,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -136,8 +138,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -156,13 +158,22 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
+
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
@@ -171,13 +182,14 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictTypeNames[type],
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -245,17 +257,25 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
+ char *updmsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -266,13 +286,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -282,47 +303,65 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ updmsg = "Could not find the row to be updated";
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("%s, and the UPDATE cannot be converted to an INSERT, thus skipping the remote changes."),
+ updmsg);
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("%s, and the UPDATE cannot be converted to an INSERT, thus raising the error."),
+ updmsg);
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("%s, thus converting the UPDATE to INSERT and %s"),
+ updmsg, applymsg);
+ else
+ appendStringInfo(&err_detail, _("%s, %s"),
+ updmsg, applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -646,6 +685,73 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
return type;
}
+/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictTypeNames[type]));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ tuple, Anum_pg_subscription_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ /*
+ * A full tuple cannot be created if any column contains a toast value.
+ * Columns with toast values are marked as LOGICALREP_COLUMN_UNCHANGED.
+ */
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
/*
* Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
* SUBSCRIPTION commands.
@@ -913,3 +1019,47 @@ RemoveSubConflictResolvers(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver for the given conflict type and subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
+ LogicalRepTupleData *newtup, bool *apply_remote)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictTypeNames[type]);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 925dff9cc4..abda71a62b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,37 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected, update the conflicting tuple by converting
+ * the remote INSERT to an UPDATE. Note that conflictslot will have the
+ * conflicting tuple only if the resolver is in favor of applying the
+ * changes, otherwise it will be NULL.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2705,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2728,48 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_ORIGIN_DIFFERS,
+ localrel, NULL,
+ &apply_remote);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2779,29 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(MySubscription->oid, CT_UPDATE_MISSING,
+ localrel, newtup, &apply_remote);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
+ }
}
/* Cleanup. */
@@ -2848,6 +2914,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2931,50 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_DELETE_ORIGIN_DIFFERS,
+ localrel, NULL, &apply_remote);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(MySubscription->oid, CT_DELETE_MISSING,
+ localrel, NULL, &apply_remote);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3021,19 +3108,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3059,6 +3148,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3071,47 +3163,87 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_MISSING,
+ partrel, newtup,
+ &apply_remote);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_ORIGIN_DIFFERS,
+ partrel, NULL,
+ &apply_remote);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3123,23 +3255,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3180,12 +3341,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3201,22 +3370,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebff00..f3909759f8 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c5865a1d58..0680ef1d4e 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -11,6 +11,7 @@
#include "nodes/execnodes.h"
#include "parser/parse_node.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -91,12 +92,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolvers(Oid subId, List *resolvers);
extern void RemoveSubConflictResolvers(Oid confid);
@@ -109,5 +112,8 @@ extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
const char *conflict_resolver);
extern List *GetDefaultConflictResolvers(void);
extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
-
+extern ConflictResolver GetConflictResolver(Oid subid, ConflictType type,
+ Relation localrel,
+ LogicalRepTupleData *newtup,
+ bool *apply_remote);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7d61..00ade29b02 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100644
index 0000000000..86e3ad1408
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,877 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test the conflict detection and resolution in logical replication
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp enabled
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT conftype, confres FROM pg_subscription_conflict ORDER BY conftype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=error/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing, resolution=skip/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+# Test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+# Test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, and the UPDATE cannot be converted to an INSERT, thus skipping the remote changes./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# Test the apply part
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_error/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+# Test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+#################################################
+# Partition table tests for UPDATE conflicts
+#################################################
+
+# Create partitioned table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);"
+);
+
+# Create similar table on subscriber but with partitions
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text) partition by range (b);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);
+ CREATE TABLE conf_tab_part_1 PARTITION OF conf_tab_part FOR VALUES FROM (MINVALUE) TO (100);
+ CREATE TABLE conf_tab_part_2 PARTITION OF conf_tab_part FOR VALUES FROM (101) TO (MAXVALUE);"
+);
+
+# Setup logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_part FOR TABLE conf_tab_part with (publish_via_partition_root=true);"
+);
+
+# Create the subscription
+$appname = 'sub_part';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=1);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is applied to the first partition");
+
+# Create a conflicting update which also changes the partition
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=101, data='frompubnew_p2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=1);");
+
+is($result, qq(101|frompubnew_p2),
+ "update from remote is the second partition");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept on the first partition");
+
+# Create a conflicting update which also changes the partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=2);");
+
+is($result, qq(1|fromsub),
+ "update from local is kept on the first partition");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=3);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is converted to insert in the first partition");
+
+# Test the update which also changes the partition
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=103, data='frompubnew_p2' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b,data from conf_tab_part WHERE (a=3);");
+
+is($result, '103|frompubnew_p2',
+ "update from remote is converted to insert in the second partition");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on first partition is skipped on the subscriber");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on second partition is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_2\": conflict=update_missing, resolution=error/,
+ $log_offset);
+
+done_testing();
--
2.34.1
v16-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v16-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From 4ea75d24cff71694daf978fe68d3f9d04127cdff Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 10 Oct 2024 15:45:28 +0530
Subject: [PATCH v16 3/4] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 40 +++-
src/backend/replication/logical/worker.c | 104 +++++++--
src/include/executor/executor.h | 3 +-
.../subscription/t/034_conflict_resolver.pl | 201 ++++++++++++++++++
4 files changed, 324 insertions(+), 24 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 27d6e98426..216e57e49f 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,7 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +497,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -543,7 +543,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid))
{
RepOriginId origin;
TimestampTz committs;
@@ -564,7 +564,7 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
* tuple information in conflictslot.
*/
static bool
-has_conflicting_tuple(Oid subid, ConflictType type,
+has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
ResultRelInfo *resultRelInfo, EState *estate,
TupleTableSlot *slot, TupleTableSlot **conflictslot)
{
@@ -573,8 +573,11 @@ has_conflicting_tuple(Oid subid, ConflictType type,
Relation rel = resultRelInfo->ri_RelationDesc;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
- /* ASSERT if called for any conflict type other than insert_exists */
- Assert(type == CT_INSERT_EXISTS);
+ /*
+ * ASSERT if called for any conflict type other than insert_exists or
+ * update_exists
+ */
+ Assert(type == CT_INSERT_EXISTS || type == CT_UPDATE_EXISTS);
/*
* Get the configured resolver and determine if remote changes should be
@@ -594,7 +597,7 @@ has_conflicting_tuple(Oid subid, ConflictType type,
{
/* Return to caller for resolutions if any conflict is found */
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid))
{
RepOriginId origin;
TimestampTz committs;
@@ -651,6 +654,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -673,8 +677,9 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* inserted tuple if a conflict is detected after insertion with a
* non-default resolution set.
*/
- if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, resultRelInfo,
- estate, slot, &(*conflictslot)))
+ if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, &invalidItemPtr,
+ resultRelInfo, estate, slot,
+ &(*conflictslot)))
return;
/* OK, store the tuple and create index entries for it */
@@ -732,7 +737,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -775,6 +781,20 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Check for conflict and return to caller for resolution, if found.
+ *
+ * XXX In case there are no conflicts, a non-default 'update_exists'
+ * resolver adds overhead by performing an extra scan here. However,
+ * this approach avoids the extra work needed to rollback/delete the
+ * updated tuple if a conflict is detected after update with a
+ * non-default resolution set.
+ */
+ if (has_conflicting_tuple(subid, CT_UPDATE_EXISTS, tid,
+ resultRelInfo, estate, slot,
+ &(*conflictslot)))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index abda71a62b..3579bd04f3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2518,8 +2519,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2671,7 +2673,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2696,14 +2699,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2711,10 +2713,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2756,6 +2759,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2768,7 +2773,52 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ *
+ * If the local table contains multiple unique constraint columns
+ * and the conflict resolution strategy favors applying the remote
+ * changes, then a remote INSERT or UPDATE could trigger multiple
+ * update_exists conflicts. Each conflict will be detected and
+ * resolved in sequence (recursively), possibly resulting in the
+ * deletion of multiple local rows.
+ *
+ * Consider the scenario: A table where all columns have unique
+ * indexes, and the first column is the replica identity. The
+ * publisher has (1, 1, 1) while the subscriber has three rows:
+ * (1, 1, 1), (2, 2, 2), and (3, 3, 3). The publisher updates (1,
+ * 1, 1) to (1, 2, 3).
+ *
+ * TThe current logic works as follows: We find the row (1, 1, 1)
+ * and try to update it to (1, 2, 3), but find a conflict with the
+ * tuple (2, 2, 2). We delete (1, 1, 1), and then try to update
+ * (2, 2, 2) to (1, 2, 3), but encounter another conflict with (3,
+ * 3, 3). We then delete (2, 2, 2) and try to update (3, 3, 3) to
+ * (1, 2, 3).
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3269,6 +3319,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3276,7 +3328,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f3909759f8..291d5dde36 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 86e3ad1408..f7c8abcecf 100644
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -156,6 +156,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=error/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -874,4 +974,105 @@ $node_subscriber->wait_for_log(
qr/ERROR: conflict detected on relation \"public.conf_tab_part_2\": conflict=update_missing, resolution=error/,
$log_offset);
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (2,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (3,1,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (4,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (5,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (6,1,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=1);");
+
+is($result, 'frompub', "update from remote on partition is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from remote on partition is skipped");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=error/,
+ $log_offset);
+
done_testing();
--
2.34.1
v16-0004-Implements-last_update_wins-conflict-resolver.patchapplication/octet-stream; name=v16-0004-Implements-last_update_wins-conflict-resolver.patchDownload
From 8d8a69c1b77d9d0d569c4ac94c7ed8ffc54d4860 Mon Sep 17 00:00:00 2001
From: Nisha Moond <nisha.moond412@gmail.com>
Date: Thu, 10 Oct 2024 15:47:18 +0530
Subject: [PATCH v16 4/4] Implements last_update_wins conflict resolver.
This resolver is applicable for conflict types: insert_exists, update_exists,
update_origin_differs and delete_origin_differs.
For these conflicts, when the resolver is set to last_update_wins,
the timestamps of the remote and local conflicting tuple are compared to
determine whether to apply or ignore the remote changes.
The GUC track_commit_timestamp must be enabled to support this resolver.
Since conflict resolution for two phase commit transactions using
prepare-timestamp can result in data divergence, this patch restricts
enabling both two_phase and the last_update_wins resolver together
for a subscription.
The patch also restrict starting a parallel apply worker if resolver is set
to last_update_wins for any conflict type.
---
src/backend/commands/subscriptioncmds.c | 26 +++-
src/backend/executor/execReplication.c | 10 +-
.../replication/logical/applyparallelworker.c | 13 ++
src/backend/replication/logical/conflict.c | 134 ++++++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 32 ++--
src/include/replication/conflict.h | 9 +-
src/include/replication/origin.h | 1 +
.../subscription/t/034_conflict_resolver.pl | 138 +++++++++++++++++-
9 files changed, 333 insertions(+), 31 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 54d8ad9554..b21387ff58 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -601,7 +601,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
/* Parse and get conflict resolvers list. */
conflict_resolvers =
- ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true);
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true, opts.twophase);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1340,6 +1340,20 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * ParseAndGetSubConflictResolvers() comments is
+ * implemented.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable \"%s\" when a time based resolver is configured",
+ "two_phase")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1593,15 +1607,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
{
List *conflict_resolvers = NIL;
+ bool sub_twophase = false;
+
+ if (sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ sub_twophase = true;
/*
* Get the list of conflict types and resolvers and validate
* them.
*/
- conflict_resolvers = ParseAndGetSubConflictResolvers(
- pstate,
+ conflict_resolvers = ParseAndGetSubConflictResolvers(pstate,
stmt->resolvers,
- false);
+ false,
+ sub_twophase);
/*
* Update the conflict resolvers for the corresponding
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 216e57e49f..8fae4078e6 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -583,7 +583,7 @@ has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
* Get the configured resolver and determine if remote changes should be
* applied.
*/
- resolver = GetConflictResolver(subid, type, rel, NULL, &apply_remote);
+ resolver = GetConflictResolver(subid, type, rel, NULL, NULL, NULL);
/*
* Proceed to find conflict if the resolver is set to a non-default value;
@@ -604,6 +604,14 @@ has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
TransactionId xmin;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(subid, type, rel, *conflictslot,
+ NULL, &apply_remote);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c5e4..6cbfab0097 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,18 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Do not start a new parallel worker if 'last_update_wins' is configured
+ * for any conflict type, as we need the commit timestamp in the
+ * beginning.
+ *
+ * XXX: To lift this restriction, we could write the changes to a file
+ * when a conflict is detected, and then at the commit time, let the
+ * remaining changes be applied by the apply worker.
+ */
+ if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 8309a70883..61d009ade7 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -43,6 +43,7 @@ static const char *const ConflictTypeNames[] = {
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -73,12 +74,12 @@ static const char *const ConflictResolverNames[] = {
* friendly name for a resolver and thus has been added here.
*/
static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS + 1] = {
- [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
- [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
- [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
+ [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, -1},
+ [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_LAST_UPDATE_WINS, -1},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, CR_LAST_UPDATE_WINS, -1},
[CT_UPDATE_MISSING] = {CR_SKIP, CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR, CR_ERROR, -1},
[CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
- [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, CR_LAST_UPDATE_WINS, -1}
};
StaticAssertDecl(lengthof(ConflictTypeResolverMap) == CONFLICT_NUM_TYPES,
@@ -382,6 +383,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -682,6 +688,14 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
conflict_resolver,
conflict_type));
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -725,6 +739,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
return resolver;
}
+/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
@@ -763,8 +813,8 @@ can_create_full_tuple(Relation localrel,
* conflict types that have not been explicitly defined by the user.
*/
List *
-ParseAndGetSubConflictResolvers(ParseState *pstate,
- List *stmtresolvers, bool add_defaults)
+ParseAndGetSubConflictResolvers(ParseState *pstate, List *stmtresolvers,
+ bool add_defaults, bool sub_twophase)
{
List *SeenTypes = NIL;
List *res = NIL;
@@ -786,6 +836,23 @@ ParseAndGetSubConflictResolvers(ParseState *pstate,
ValidateConflictTypeAndResolver(defel->defname,
resolver);
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ *
+ * XXX: An alternative solution idea is that if a conflict is detected
+ * and the resolution strategy is last_update_wins, then start writing
+ * all the changes to a file similar to what we do for streaming mode.
+ * Once commit_prepared arrives, we will read and apply the changes.
+ */
+ if ((pg_strcasecmp(resolver, "last_update_wins") == 0) &&
+ sub_twophase)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver when \"%s\" is enabled; these options are mutually exclusive",
+ "last_update_wins", "two_phase")));
+
/* Add the conflict type to the list of seen types */
SeenTypes = lappend(SeenTypes, makeString(defel->defname));
@@ -1028,14 +1095,30 @@ RemoveSubConflictResolvers(Oid subid)
*/
ConflictResolver
GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
- LogicalRepTupleData *newtup, bool *apply_remote)
+ TupleTableSlot *conflictslot, LogicalRepTupleData *newtup,
+ bool *apply_remote)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, simply return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(conflictslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1063,3 +1146,40 @@ GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index baf696d8e6..3094030103 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -159,6 +159,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3579bd04f3..291e79a6ac 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1006,6 +1006,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -2741,7 +2747,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
resolver = GetConflictResolver(MySubscription->oid,
CT_UPDATE_ORIGIN_DIFFERS,
- localrel, NULL,
+ localrel, localslot, NULL,
&apply_remote);
/* Store the new tuple for conflict reporting */
@@ -2834,7 +2840,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* UPDATE to INSERT and apply the change.
*/
resolver = GetConflictResolver(MySubscription->oid, CT_UPDATE_MISSING,
- localrel, newtup, &apply_remote);
+ localrel, localslot, newtup,
+ &apply_remote);
ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
remoteslot, NULL, newslot, InvalidOid,
@@ -2989,7 +2996,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
{
resolver = GetConflictResolver(MySubscription->oid,
CT_DELETE_ORIGIN_DIFFERS,
- localrel, NULL, &apply_remote);
+ localrel, localslot, NULL,
+ &apply_remote);
ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
resolver, remoteslot, localslot, NULL,
@@ -3017,7 +3025,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* configured, either skip and log a message or emit an error.
*/
resolver = GetConflictResolver(MySubscription->oid, CT_DELETE_MISSING,
- localrel, NULL, &apply_remote);
+ localrel, localslot, NULL,
+ &apply_remote);
/* Resolver is set to skip, thus report the conflict and skip */
if (!apply_remote)
@@ -3214,8 +3223,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
resolver = GetConflictResolver(MySubscription->oid,
- CT_UPDATE_MISSING,
- partrel, newtup,
+ CT_UPDATE_MISSING, partrel,
+ localslot, newtup,
&apply_remote);
/*
@@ -3259,7 +3268,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
resolver = GetConflictResolver(MySubscription->oid,
CT_UPDATE_ORIGIN_DIFFERS,
- partrel, NULL,
+ partrel, localslot, NULL,
&apply_remote);
ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
@@ -4790,6 +4799,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4830,10 +4840,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0680ef1d4e..9eca7fccbe 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "parser/parse_node.h"
#include "replication/logicalrelation.h"
@@ -66,6 +67,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it cannot be applied */
CR_APPLY_OR_SKIP,
@@ -105,7 +109,8 @@ extern void SetSubConflictResolvers(Oid subId, List *resolvers);
extern void RemoveSubConflictResolvers(Oid confid);
extern List *ParseAndGetSubConflictResolvers(ParseState *pstate,
List *stmtresolvers,
- bool add_defaults);
+ bool add_defaults,
+ bool sub_twophase);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType ValidateConflictType(const char *conflict_type);
extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
@@ -114,6 +119,8 @@ extern List *GetDefaultConflictResolvers(void);
extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
extern ConflictResolver GetConflictResolver(Oid subid, ConflictType type,
Relation localrel,
+ TupleTableSlot *localslot,
LogicalRepTupleData *newtup,
bool *apply_remote);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
#endif
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9e76..dcbbbdf6ea 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index f7c8abcecf..f6a7258aa4 100644
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -156,6 +156,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -256,6 +284,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -309,16 +367,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -405,10 +495,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -426,7 +520,7 @@ $node_publisher->safe_psql('postgres',
"UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
$node_subscriber->wait_for_log(
- qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=last_update_wins/,
$log_offset);
# Confirm that the remote update overrides the local update
@@ -435,6 +529,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
--
2.34.1
Hi Ajin/Nisha -- Here are my review comments for patch v15-0001 (code).
(AFAIK v16-0001 is the same as v15-0001, so this review is up to date)
Please also see the "nits" attachment to this post, which has many
more review comments of a more cosmetic nature.
======
src/backend/replication/logical/conflict.c
1.
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
Add missing static assertions:
StaticAssertDecl(lengthof(ConflictTypeNames) == CONFLICT_NUM_TYPES,
"array length mismatch");
StaticAssertDecl(lengthof(ConflictResolverNames) == CONFLICT_NUM_TYPES,
"array length mismatch");
~~~
2.
+#define CONFLICT_TYPE_MAX_RESOLVERS 4
Now unused. Remove this.
~~~
3.
+static const int ConflictTypeResolverMap[][CONFLICT_NUM_RESOLVERS + 1] = {
+ [CT_INSERT_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_EXISTS] = {CR_ERROR, CR_APPLY_REMOTE, CR_KEEP_LOCAL, -1},
+ [CT_UPDATE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1},
+ [CT_UPDATE_MISSING] = {CR_SKIP, CR_APPLY_OR_SKIP, CR_APPLY_OR_ERROR,
CR_ERROR, -1},
+ [CT_DELETE_MISSING] = {CR_SKIP, CR_ERROR, -1},
+ [CT_DELETE_ORIGIN_DIFFERS] = {CR_APPLY_REMOTE, CR_KEEP_LOCAL, CR_ERROR, -1}
+};
If you are planning to keep this implementation of the Map, then that
-1 end of List marker is critical to the logic for the scanner part to
know when to stop looking for valid resolvers. So, I think you should
add another comment about that.
~~~
4.
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to
disabled track_commit_timestamp"),
+ errdetail("Conflicts update_origin_differs and delete_origin_differs
cannot be "
+ "detected, and the origin and commit timestamp for the local row "
+ "will not be logged."));
+}
+
The "could be incomplete" wording seems vague to me. Why now just say
the WARNING reason in the errmsg?
SUGGESTION:
Conflicts types 'update_origin_differs' and 'delete_origin_differs'
cannot be detected unless "track_commit_timestamp" is enabled.
~~~
ParseAndGetSubConflictResolvers:
5.
+{
+ List *SeenTypes = NIL;
+ List *res = NIL;
All the list/string processing to determine if we have "seen" a
conflict type before looks inefficient to me. I had already
demonstrated in a previous review how this can be ditched in favour of
a simple boolean array. Again, I have implemented this in the NITPICKS
attachment to show how it can be implemented.
~~~
6.
+/*
+ * Get the list of conflict types and their corresponding default resolvers.
+ */
+List *
+GetDefaultConflictResolvers()
I'm wondering if it is worthwhile caching this default list. It will
never change, so if you cache it then you don't need to recalculate it
on subsequent calls.
======
src/bin/pg_dump/pg_dump.c
7.
Where are the test cases for the CONFLICT RESOLVER dump code?
~~~
getSubscriptions:
8.
+ ntups,
+ ntuples;
These var names are too similar. I can't tell them apart. Please
rename the new one (e.g. 'conf_ntuples').
~
9.
+ /* Initialize pointers in the list to NULL */
+ subinfo[i].conflict_resolver = (SimplePtrList)
+ {
+ 0
+ };
+
I didn't find anything else like this in PG source. IMO, it is better
to initialize both members explicitly to make it clear what this code
is actually doing. Or maybe memset() it.
SUGGESTION:
subinfo[i].conflict_resolver = (SimplePtrList)
{
.head = NULL, .tail = NULL
};
~~~
dumpSubscription:
+ /* Add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ bool first_resolver = true;
10a.
AFAICT this code is misplaced/broken. In the CREATE SUBSCRIPTION
syntax, the CONFLICT RESOLVER should come *before* any WITH clause, so
I think this is doomed to give a syntax error. The (missing) test
cases would have found this.
10b.
And, I expect when you fix that clause ordering then there will be
other fixes needed to correctly handle the closing parenthesis ')'.
~~~
11.
+/*
+ * destroyConflictResolverList - frees up the SimplePtrList containing
+ * cells pointing to struct ConflictTypeResolver nodes.
+ */
+static void
+destroyConcflictResolverList(SimplePtrList *conflictlist)
+{
+ SimplePtrListCell *cell = NULL;
+
+ /* Free the list items */
+ for (cell = conflictlist->head; cell; cell = cell->next)
+ pfree((ConflictTypeResolver *) cell->ptr);
+
+ /* Destroy the pointer list */
+ simple_ptr_list_destroy(conflictlist);
+}
Hmm. Is this even needed? AFAICT this entire function seems to be
doing the same as just calling simple_ptr_list_destroy(conflictlist)
directly.
======
src/bin/pg_dump/pg_dump.h
12.
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
This should be renamed (e.g. _ConflictTypeResolver or similar) with
underscore for consistency with other dump typedefs
~~~
13.
char *subfailover;
+ SimplePtrList conflict_resolver;
} SubscriptionInfo;
Rename this field to 'conflict_resolvers' (plural). Also, a comment
might help to say what the list elements are.
======
MISSING src/bin/psql/tab-complete.c
14.
Where is the tab-completion implementation for all the new syntax of
the v15 patch?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
Attachments:
PS_NITPICKS_CDR_V150001_CODE.txttext/plain; charset=US-ASCII; name=PS_NITPICKS_CDR_V150001_CODE.txtDownload
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 54d8ad9..1261d30 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -729,7 +729,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
- /* Update the Conflict Resolvers in pg_subscription_conflict */
+ /* Update the conflict resolvers in pg_subscription_conflict. */
SetSubConflictResolvers(subid, conflict_resolvers);
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
@@ -1598,10 +1598,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
* Get the list of conflict types and resolvers and validate
* them.
*/
- conflict_resolvers = ParseAndGetSubConflictResolvers(
- pstate,
- stmt->resolvers,
- false);
+ conflict_resolvers =
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, false);
/*
* Update the conflict resolvers for the corresponding
@@ -1882,7 +1880,7 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
- /* Remove any associated conflict resolvers */
+ /* Remove any associated conflict resolvers. */
RemoveSubConflictResolvers(subid);
/* Remove the origin tracking if exists. */
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 28591c9..68e0fa9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -10898,6 +10898,7 @@ AlterSubscriptionStmt:
$$ = (Node *) n;
}
;
+
conflict_type:
Sconst { $$ = $1; }
;
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index eb46e39..a20a86e 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -40,6 +40,9 @@ static const char *const ConflictTypeNames[] = {
[CT_DELETE_MISSING] = "delete_missing"
};
+StaticAssertDecl(lengthof(ConflictTypeNames) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
+
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
@@ -49,7 +52,8 @@ static const char *const ConflictResolverNames[] = {
[CR_ERROR] = "error"
};
-#define CONFLICT_TYPE_MAX_RESOLVERS 4
+StaticAssertDecl(lengthof(ConflictResolverNames) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
/*
* Valid conflict resolvers for each conflict type.
@@ -562,7 +566,7 @@ conf_detection_check_prerequisites(void)
* Validate the conflict type and return the corresponding ConflictType enum.
*/
ConflictType
-ValidateConflictType(const char *conflict_type)
+ValidateConflictType(const char *conflict_type_name)
{
ConflictType type;
bool valid = false;
@@ -570,7 +574,7 @@ ValidateConflictType(const char *conflict_type)
/* Check conflict type validity */
for (type = 0; type < CONFLICT_NUM_TYPES; type++)
{
- if (pg_strcasecmp(ConflictTypeNames[type], conflict_type) == 0)
+ if (pg_strcasecmp(ConflictTypeNames[type], conflict_type_name) == 0)
{
valid = true;
break;
@@ -580,30 +584,30 @@ ValidateConflictType(const char *conflict_type)
if (!valid)
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("%s is not a valid conflict type", conflict_type));
+ errmsg("%s is not a valid conflict type", conflict_type_name));
return type;
}
/*
* Validate the conflict type and resolver. It returns an enum ConflictType
- * corresponding to the conflict type string passed by the caller.
+ * corresponding to the conflict type name passed by the caller.
*/
ConflictType
-ValidateConflictTypeAndResolver(const char *conflict_type,
- const char *conflict_resolver)
+ValidateConflictTypeAndResolver(const char *conflict_type_name,
+ const char *conflict_resolver_name)
{
ConflictType type;
ConflictResolver resolver;
bool valid = false;
/* Validate conflict type */
- type = ValidateConflictType(conflict_type);
+ type = ValidateConflictType(conflict_type_name);
/* Validate the conflict resolver name */
for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
{
- if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver_name) == 0)
{
valid = true;
break;
@@ -614,7 +618,7 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
{
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("%s is not a valid conflict resolver", conflict_resolver));
+ errmsg("%s is not a valid conflict resolver", conflict_resolver_name));
}
/* Reset */
@@ -640,12 +644,13 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("%s is not a valid conflict resolver for conflict type %s",
- conflict_resolver,
- conflict_type));
+ conflict_resolver_name,
+ conflict_type_name));
return type;
}
+
/*
* Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
* SUBSCRIPTION commands.
@@ -660,36 +665,33 @@ List *
ParseAndGetSubConflictResolvers(ParseState *pstate,
List *stmtresolvers, bool add_defaults)
{
- List *SeenTypes = NIL;
+ bool seen_type[CONFLICT_NUM_TYPES] = {0};
List *res = NIL;
- /* Loop through the user provided resolvers */
+ /* Loop through the user provided resolvers. */
foreach_ptr(DefElem, defel, stmtresolvers)
{
- char *resolver;
+ char *resolver_name;
ConflictTypeResolver *conftyperesolver = NULL;
+ ConflictType conflict_type;
- /* Check if the conflict type has already been seen */
- if (list_member(SeenTypes, makeString(defel->defname)))
- errorConflictingDefElem(defel, pstate);
+ /* Validate the conflict type and resolver. */
+ resolver_name = defGetString(defel);
+ resolver_name = downcase_truncate_identifier(resolver_name, strlen(resolver_name), false);
+ conflict_type = ValidateConflictTypeAndResolver(defel->defname, resolver_name);
- /* Validate the conflict type and resolver */
- resolver = defGetString(defel);
- resolver = downcase_truncate_identifier(
- resolver, strlen(resolver), false);
- ValidateConflictTypeAndResolver(defel->defname,
- resolver);
-
- /* Add the conflict type to the list of seen types */
- SeenTypes = lappend(SeenTypes, makeString(defel->defname));
+ /* Error if the conflict type has already been seen, else flag it is as seen. */
+ if (seen_type[conflict_type])
+ errorConflictingDefElem(defel, pstate);
+ seen_type[conflict_type] = true;
conftyperesolver = palloc(sizeof(ConflictTypeResolver));
conftyperesolver->conflict_type_name = defel->defname;
- conftyperesolver->conflict_resolver_name = resolver;
+ conftyperesolver->conflict_resolver_name = resolver_name;
res = lappend(res, conftyperesolver);
}
- /* Once validation is complete, warn users if prerequisites are not met */
+ /* Once validation is complete, warn users if prerequisites are not met. */
if (stmtresolvers)
conf_detection_check_prerequisites();
@@ -701,11 +703,10 @@ ParseAndGetSubConflictResolvers(ParseState *pstate,
{
for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
{
- ConflictTypeResolver *conftyperesolver = NULL;
-
- if (!list_member(SeenTypes, makeString((char *) ConflictTypeNames[i])))
+ if (!seen_type[i])
{
ConflictResolver def_resolver = ConflictTypeResolverMap[i][0];
+ ConflictTypeResolver *conftyperesolver;
conftyperesolver = palloc(sizeof(ConflictTypeResolver));
conftyperesolver->conflict_type_name = ConflictTypeNames[i];
@@ -715,8 +716,6 @@ ParseAndGetSubConflictResolvers(ParseState *pstate,
}
}
- list_free_deep(SeenTypes);
-
return res;
}
@@ -726,7 +725,11 @@ ParseAndGetSubConflictResolvers(ParseState *pstate,
List *
GetDefaultConflictResolvers()
{
- List *res = NIL;
+ static List *res = NIL;
+
+ /* The defaults are always same, so return the same list. */
+ if (res)
+ return res;
for (ConflictType type = 0; type < CONFLICT_NUM_TYPES; type++)
{
@@ -756,13 +759,9 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
Datum values[Natts_pg_subscription_conflict];
bool nulls[Natts_pg_subscription_conflict];
bool replaces[Natts_pg_subscription_conflict];
- HeapTuple oldtup;
- HeapTuple newtup = NULL;
Relation pg_subscription_conflict;
- char *cur_conflict_res;
- Datum datum;
- /* Prepare to update a tuple */
+ /* Prepare to update a tuple. */
memset(nulls, false, sizeof(nulls));
memset(replaces, false, sizeof(replaces));
memset(values, 0, sizeof(values));
@@ -771,7 +770,12 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
{
- /* Set up subid and conflict_type to search in cache */
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Set up subid and conflict_type to search in cache. */
values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
values[Anum_pg_subscription_conflict_conftype - 1] =
CStringGetTextDatum(conftyperesolver->conflict_type_name);
@@ -795,7 +799,7 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
if (pg_strcasecmp(cur_conflict_res,
conftyperesolver->conflict_resolver_name) != 0)
{
- /* Update the new resolver */
+ /* Update the new resolver. */
values[Anum_pg_subscription_conflict_confres - 1] =
CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
replaces[Anum_pg_subscription_conflict_confres - 1] = true;
@@ -818,29 +822,28 @@ UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
* Reset the conflict resolver for this conflict type to its default setting.
*/
void
-ResetSubConflictResolver(Oid subid, char *conflict_type)
+ResetSubConflictResolver(Oid subid, char *conflict_type_name)
{
- ConflictType idx;
- ConflictTypeResolver *conflictResolver = NULL;
- List *conflictresolver_list = NIL;
-
- /* Validate the conflict type and get the index */
- idx = ValidateConflictType(conflict_type);
- conflictResolver = palloc(sizeof(ConflictTypeResolver));
- conflictResolver->conflict_type_name = conflict_type;
-
- /* Get the default resolver for this conflict_type */
- conflictResolver->conflict_resolver_name =
- ConflictResolverNames[ConflictTypeResolverMap[idx][0]];
-
- /* Create a list of conflict resolvers and update in catalog */
- conflictresolver_list = lappend(conflictresolver_list, conflictResolver);
- UpdateSubConflictResolvers(conflictresolver_list, subid);
-
+ ConflictType conflict_type;
+ ConflictTypeResolver *ctr = NULL;
+ List *ctr_list = NIL;
+
+ /* Validate the conflict type name and get the conflict type. */
+ conflict_type = ValidateConflictType(conflict_type_name);
+ ctr = palloc(sizeof(ConflictTypeResolver));
+ ctr->conflict_type_name = conflict_type_name;
+
+ /* Get the default resolver for this conflict type. */
+ ctr->conflict_resolver_name =
+ ConflictResolverNames[ConflictTypeResolverMap[conflict_type][0]];
+
+ /* Create a (one-element) list of ConflictTypeResolver's and update the catalog. */
+ ctr_list = lappend(ctr_list, ctr);
+ UpdateSubConflictResolvers(ctr_list, subid);
}
/*
- * Set Conflict Resolvers on the subscription
+ * Set Conflict Resolvers on the subscription.
*/
void
SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
@@ -848,17 +851,18 @@ SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
Relation pg_subscription_conflict;
Datum values[Natts_pg_subscription_conflict];
bool nulls[Natts_pg_subscription_conflict];
- HeapTuple newtup = NULL;
- Oid conflict_oid;
pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
- /* Prepare to update a tuple */
+ /* Prepare to update a tuple. */
memset(nulls, false, sizeof(nulls));
- /* Iterate over the list of resolvers */
+ /* Iterate over the list of resolvers. */
foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
{
+ HeapTuple newtup = NULL;
+ Oid conflict_oid;
+
values[Anum_pg_subscription_conflict_confsubid - 1] =
ObjectIdGetDatum(subId);
values[Anum_pg_subscription_conflict_conftype - 1] =
@@ -866,7 +870,7 @@ SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
values[Anum_pg_subscription_conflict_confres - 1] =
CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
- /* Get a new oid and update the tuple into catalog */
+ /* Get a new oid and update the tuple into catalog. */
conflict_oid = GetNewOidWithIndex(pg_subscription_conflict,
SubscriptionConflictOidIndexId,
Anum_pg_subscription_conflict_oid);
@@ -882,7 +886,7 @@ SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
}
/*
- * Remove the subscription conflict resolvers for the subscription id
+ * Remove the subscription conflict resolvers for the subscription id.
*/
void
RemoveSubConflictResolvers(Oid subid)
@@ -906,7 +910,7 @@ RemoveSubConflictResolvers(Oid subid)
scan = table_beginscan_catalog(rel, 1, skey);
- /* Iterate through the tuples and delete them */
+ /* Iterate through the tuples and delete them. */
while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
CatalogTupleDelete(rel, &tup->t_self);
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 572eb52..860422e 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -395,7 +395,7 @@ static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
static bool forcePartitionRootLoad(const TableInfo *tbinfo);
static void read_dump_filters(const char *filename, DumpOptions *dopt);
-static bool is_default_resolver(const char *confType, const char *confRes);
+static bool is_default_resolver(const char *conf_type_name, const char *conf_resolver_name);
static void destroyConcflictResolverList(SimplePtrList *list);
@@ -4832,9 +4832,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
- PQExpBuffer confQuery;
+ PQExpBuffer conf_query;
PGresult *res;
- PGresult *confRes;
+ PGresult *conf_res;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4855,9 +4855,8 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- j,
ntups,
- ntuples;
+ conf_ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5018,36 +5017,36 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
- /* Populate conflict type fields using the new query */
- confQuery = createPQExpBuffer();
- appendPQExpBuffer(confQuery,
+ /* Populate conflict type fields using the new query. */
+ conf_query = createPQExpBuffer();
+ appendPQExpBuffer(conf_query,
"SELECT conftype, confres FROM pg_catalog.pg_subscription_conflict "
- "WHERE confsubid = %u order by conftype;", subinfo[i].dobj.catId.oid);
- confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+ "WHERE confsubid = %u ORDER BY conftype;", subinfo[i].dobj.catId.oid);
+ conf_res = ExecuteSqlQuery(fout, conf_query->data, PGRES_TUPLES_OK);
- ntuples = PQntuples(confRes);
+ conf_ntuples = PQntuples(conf_res);
- /* Initialize pointers in the list to NULL */
- subinfo[i].conflict_resolver = (SimplePtrList)
+ /* Initialize pointers in the list to NULL. */
+ subinfo[i].conflict_resolvers = (SimplePtrList)
{
- 0
+ .head = NULL, .tail = NULL
};
- /* Store conflict types and resolvers from the query result in subinfo */
- for (j = 0; j < ntuples; j++)
+ /* Store conflict types and resolvers from the query result in subinfo. */
+ for (int j = 0; j < conf_ntuples; j++)
{
- /* Create the ConflictTypeResolver node */
- ConflictTypeResolver *conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ /* Create the _ConflictTypeResolver node. */
+ _ConflictTypeResolver *ctr = palloc(sizeof(_ConflictTypeResolver));
- conftyperesolver->conflict_type = pg_strdup(PQgetvalue(confRes, j, 0));
- conftyperesolver->resolver = pg_strdup(PQgetvalue(confRes, j, 1));
+ ctr->conflict_type_name = pg_strdup(PQgetvalue(conf_res, j, 0));
+ ctr->conflict_resolver_name = pg_strdup(PQgetvalue(conf_res, j, 1));
- /* Append the node to subinfo's list */
- simple_ptr_list_append(&subinfo[i].conflict_resolver, conftyperesolver);
+ /* Append the node to subinfo's list. */
+ simple_ptr_list_append(&subinfo[i].conflict_resolvers, ctr);
}
- PQclear(confRes);
- destroyPQExpBuffer(confQuery);
+ PQclear(conf_res);
+ destroyPQExpBuffer(conf_query);
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
@@ -5299,26 +5298,24 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
{
bool first_resolver = true;
SimplePtrListCell *cell = NULL;
- ConflictTypeResolver *conftyperesolver = NULL;
- for (cell = subinfo->conflict_resolver.head; cell; cell = cell->next)
+ for (cell = subinfo->conflict_resolvers.head; cell; cell = cell->next)
{
- conftyperesolver = (ConflictTypeResolver *) cell->ptr;
+ _ConflictTypeResolver *ctr = (_ConflictTypeResolver *) cell->ptr;
- if (!is_default_resolver(conftyperesolver->conflict_type,
- conftyperesolver->resolver))
+ if (!is_default_resolver(ctr->conflict_type_name, ctr->conflict_resolver_name))
{
if (first_resolver)
{
appendPQExpBuffer(query, ") CONFLICT RESOLVER (%s = '%s'",
- conftyperesolver->conflict_type,
- conftyperesolver->resolver);
+ ctr->conflict_type_name,
+ ctr->conflict_resolver_name);
first_resolver = false;
}
else
appendPQExpBuffer(query, ", %s = '%s'",
- conftyperesolver->conflict_type,
- conftyperesolver->resolver);
+ ctr->conflict_type_name,
+ ctr->conflict_resolver_name);
}
}
}
@@ -5382,7 +5379,7 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
subinfo->dobj.catId, 0, subinfo->dobj.dumpId);
/* Clean-up the conflict_resolver list */
- destroyConcflictResolverList((SimplePtrList *) &subinfo->conflict_resolver);
+ destroyConcflictResolverList((SimplePtrList *) &subinfo->conflict_resolvers);
destroyPQExpBuffer(publications);
free(pubnames);
@@ -19126,7 +19123,7 @@ read_dump_filters(const char *filename, DumpOptions *dopt)
* specified conflict type.
*/
static bool
-is_default_resolver(const char *confType, const char *confRes)
+is_default_resolver(const char *conf_type_name, const char *conf_resolver_name)
{
/*
* The default resolvers for each conflict type are taken from the
@@ -19136,15 +19133,23 @@ is_default_resolver(const char *confType, const char *confRes)
* are changed.
*/
- if (strcmp(confType, "insert_exists") == 0 ||
- strcmp(confType, "update_exists") == 0)
- return strcmp(confRes, "error") == 0;
- else if (strcmp(confType, "update_missing") == 0 ||
- strcmp(confType, "delete_missing") == 0)
- return strcmp(confRes, "skip") == 0;
- else if (strcmp(confType, "update_origin_differs") == 0 ||
- strcmp(confType, "delete_origin_differs") == 0)
- return strcmp(confRes, "apply_remote") == 0;
+ if (strcmp(conf_type_name, "insert_exists") == 0)
+ return strcmp(conf_resolver_name, "error") == 0;
+
+ else if (strcmp(conf_type_name, "update_exists") == 0)
+ return strcmp(conf_resolver_name, "error") == 0;
+
+ else if (strcmp(conf_type_name, "update_missing") == 0)
+ return strcmp(conf_resolver_name, "skip") == 0;
+
+ else if (strcmp(conf_type_name, "update_origin_differs") == 0)
+ return strcmp(conf_resolver_name, "apply_remote") == 0;
+
+ else if (strcmp(conf_type_name, "delete_missing") == 0)
+ return strcmp(conf_resolver_name, "skip") == 0;
+
+ else if (strcmp(conf_type_name, "delete_origin_differs") == 0)
+ return strcmp(conf_resolver_name, "apply_remote") == 0;
return false;
}
@@ -19160,7 +19165,7 @@ destroyConcflictResolverList(SimplePtrList *conflictlist)
/* Free the list items */
for (cell = conflictlist->head; cell; cell = cell->next)
- pfree((ConflictTypeResolver *) cell->ptr);
+ pfree((_ConflictTypeResolver *) cell->ptr);
/* Destroy the pointer list */
simple_ptr_list_destroy(conflictlist);
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index bb16e8b..5c69ad9 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,12 +655,11 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
-
-typedef struct ConflictTypeResolver
+typedef struct _ConflictTypeResolver
{
- const char *conflict_type;
- const char *resolver;
-} ConflictTypeResolver;
+ const char *conflict_type_name;
+ const char *conflict_resolver_name;
+} _ConflictTypeResolver;
typedef struct _SubscriptionInfo
{
@@ -680,7 +679,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
- SimplePtrList conflict_resolver;
+ SimplePtrList conflict_resolvers;
} SubscriptionInfo;
/*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8599f22..ee37043 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6628,7 +6628,7 @@ describeSubscriptions(const char *pattern, bool verbose)
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
- /* Add conflict resolvers information from pg_subscription_conflict */
+ /* Add conflict resolvers information from pg_subscription_conflict. */
if (pset.sversion >= 180000)
appendPQExpBuffer(&buf,
", (SELECT string_agg(conftype || ' = ' || confres, ', ' \n"
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 9791141..9c3a950 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -411,12 +411,12 @@ ERROR: foo is not a valid conflict resolver
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
--- fail - duplicate conflict type
+-- fail - duplicate conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
ERROR: conflicting or redundant options
LINE 1: ... CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exi...
^
--- creating subscription with no explicit conflict resolvers should
+-- ok - creating subscription with no explicit conflict resolvers should
-- configure default conflict resolvers
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
WARNING: subscription was created, but is not connected
@@ -444,18 +444,18 @@ HINT: To initiate replication, you must manually create the replication slot, e
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = keep_local, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
--- fail - altering with invalid conflict type
+-- fail - alter with invalid conflict type
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
ERROR: foo is not a valid conflict type
--- fail - altering with invalid conflict resolver
+-- fail - alter with invalid conflict resolver
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
ERROR: foo is not a valid conflict resolver
--- fail - altering with duplicate conflict type
+-- fail - alter with duplicate conflict types
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
ERROR: conflicting or redundant options
LINE 1: ...ONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exi...
^
--- ok - valid conflict types and resolvers
+-- ok - alter with valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
@@ -466,7 +466,7 @@ DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be det
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = skip, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = error, update_missing = skip, update_origin_differs = apply_remote
(1 row)
--- ok - valid conflict types and resolvers
+-- ok - alter with valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be detected, and the origin and commit timestamp for the local row will not be logged.
@@ -477,11 +477,11 @@ DETAIL: Conflicts update_origin_differs and delete_origin_differs cannot be det
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0 | delete_missing = error, delete_origin_differs = keep_local, insert_exists = apply_remote, update_exists = keep_local, update_missing = skip, update_origin_differs = error
(1 row)
--- fail - reset with an invalid conflit type
-ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+-- fail - reset for invalid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER FOR 'foo';
ERROR: foo is not a valid conflict type
--- ok - valid conflict type
-ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+-- ok - reset for valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER FOR 'insert_exists';
\dRs+
List of subscriptions
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN | Conflict Resolvers
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 3f08869..9d42900 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -282,10 +282,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
--- fail - duplicate conflict type
+-- fail - duplicate conflict types
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
--- creating subscription with no explicit conflict resolvers should
+-- ok - creating subscription with no explicit conflict resolvers should
-- configure default conflict resolvers
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
@@ -300,30 +300,30 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
--check if above are configured; for non specified conflict types, default resolvers should be seen
\dRs+
--- fail - altering with invalid conflict type
+-- fail - alter with invalid conflict type
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
--- fail - altering with invalid conflict resolver
+-- fail - alter with invalid conflict resolver
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
--- fail - altering with duplicate conflict type
+-- fail - alter with duplicate conflict types
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
--- ok - valid conflict types and resolvers
+-- ok - alter with valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
\dRs+
--- ok - valid conflict types and resolvers
+-- ok - alter with valid conflict types and resolvers
ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
\dRs+
--- fail - reset with an invalid conflit type
-ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+-- fail - reset for invalid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER FOR 'foo';
--- ok - valid conflict type
-ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+-- ok - reset for valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER FOR 'insert_exists';
\dRs+
On Wednesday, October 9, 2024 2:34 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Oct 9, 2024 at 8:58 AM shveta malik <shveta.malik@gmail.com>
wrote:On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond
<nisha.moond412@gmail.com> wrote:
Please find few comments on v14-patch004:
patch004:
1)
GetConflictResolver currently errors out when the resolver is last_update_wins
and track_commit_timestamp is disabled. It means every conflict resolution
with this resolver will keep on erroring out. I am not sure if we should emit
ERROR here. We do emit ERROR when someone tries to configure
last_update_wins but track_commit_timestamp is disabled. I think that should
suffice. The one in GetConflictResolver can be converted to WARNING max.What could be the side-effect if we do not emit error here? In such a case, the
local timestamp will be 0 and remote change will always win.
Is that right? If so, then if needed, we can emit a warning saying something like:
'track_commit_timestamp is disabled and thus remote change is applied
always.'Thoughts?
I think simply reporting a warning and applying remote changes without further
action could lead to data inconsistencies between nodes. Considering the
potential challenges and time required to recover from these inconsistencies, I
prefer to keep reporting errors, in which case users have an opportunity to
resolve the issue by enabling track_commit_timestamp.
Best Regards,
Hou zj
On Tue, Oct 15, 2024 at 6:00 PM Peter Smith <smithpb2250@gmail.com> wrote:
Hi Ajin/Nisha -- Here are my review comments for patch v15-0001 (code).
(AFAIK v16-0001 is the same as v15-0001, so this review is up to date)
Please also see the "nits" attachment to this post, which has many
more review comments of a more cosmetic nature.
Here's patch v17 addressing all review comments from Peter and Shveta:
Thanks Nisha for working on patches 0003 and 0004.
Patches:
- /messages/by-id/CAJpy0uAEfA9Bhi=8U6dP--Jovj4ZZK88x2upD06xV1eMEAGhxw@mail.gmail.com
- /messages/by-id/CAJpy0uAf9DugEeat0SASnch7LHyeJxsvoMrvoH08uBbOcKFSqQ@mail.gmail.com
- /messages/by-id/CAHut+PsDH-=RMsemWau+_DL1sLrMc5DjQpLTgek22UW=1jPayg@mail.gmail.com
Attachments:
v17-0004-Implements-last_update_wins-conflict-resolver.patchapplication/octet-stream; name=v17-0004-Implements-last_update_wins-conflict-resolver.patchDownload
From 351e503c9c6469c0d7b5d362c2bd032e1f044e69 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 18 Oct 2024 06:55:14 -0400
Subject: [PATCH v17 4/4] Implements last_update_wins conflict resolver.
This resolver is applicable for conflict types: insert_exists, update_exists,
update_origin_differs and delete_origin_differs.
For these conflicts, when the resolver is set to last_update_wins,
the timestamps of the remote and local conflicting tuple are compared to
determine whether to apply or ignore the remote changes.
The GUC track_commit_timestamp must be enabled to support this resolver.
Since conflict resolution for two phase commit transactions using
prepare-timestamp can result in data divergence, this patch restricts
enabling both two_phase and the last_update_wins resolver together
for a subscription.
The patch also restrict starting a parallel apply worker if resolver is set
to last_update_wins for any conflict type.
---
src/backend/commands/subscriptioncmds.c | 26 +++-
src/backend/executor/execReplication.c | 12 +-
.../replication/logical/applyparallelworker.c | 13 ++
src/backend/replication/logical/conflict.c | 137 +++++++++++++++++++-
src/backend/replication/logical/origin.c | 1 +
src/backend/replication/logical/worker.c | 32 +++--
src/include/replication/conflict.h | 9 +-
src/include/replication/origin.h | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 138 +++++++++++++++++++--
9 files changed, 338 insertions(+), 31 deletions(-)
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 54d8ad9..b21387f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -601,7 +601,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
/* Parse and get conflict resolvers list. */
conflict_resolvers =
- ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true);
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true, opts.twophase);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -1340,6 +1340,20 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
errmsg("cannot disable two_phase when prepared transactions are present"),
errhint("Resolve these transactions and try again.")));
+ /*
+ * two_phase cannot be enabled if sub has a time based
+ * resolver set, as it may result in data divergence.
+ *
+ * XXX: This restriction may be removed if the solution in
+ * ParseAndGetSubConflictResolvers() comments is
+ * implemented.
+ */
+ if (opts.twophase && CheckIfSubHasTimeStampResolver(subid))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot enable \"%s\" when a time based resolver is configured",
+ "two_phase")));
+
/* Change system catalog accordingly */
values[Anum_pg_subscription_subtwophasestate - 1] =
CharGetDatum(opts.twophase ?
@@ -1593,15 +1607,19 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
{
List *conflict_resolvers = NIL;
+ bool sub_twophase = false;
+
+ if (sub->twophasestate != LOGICALREP_TWOPHASE_STATE_DISABLED)
+ sub_twophase = true;
/*
* Get the list of conflict types and resolvers and validate
* them.
*/
- conflict_resolvers = ParseAndGetSubConflictResolvers(
- pstate,
+ conflict_resolvers = ParseAndGetSubConflictResolvers(pstate,
stmt->resolvers,
- false);
+ false,
+ sub_twophase);
/*
* Update the conflict resolvers for the corresponding
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index cb6021b..ec88af0 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -586,7 +586,7 @@ has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
* Get the configured resolver and determine if remote changes should be
* applied.
*/
- resolver = GetConflictResolver(subid, type, rel, NULL, &apply_remote);
+ resolver = GetConflictResolver(subid, type, rel, NULL, NULL, NULL);
/*
* Determine the lock mode for the conflicting tuple, if any. Take an
@@ -594,7 +594,7 @@ has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
* tuple; otherwise, take a shared lock.
*/
- if (apply_remote)
+ if (resolver == CR_LAST_UPDATE_WINS || apply_remote)
lockmode = LockTupleExclusive;
else
lockmode = LockTupleShare;
@@ -618,6 +618,14 @@ has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
TransactionId xmin;
GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+
+ /*
+ * Get the configured resolver and determine if remote changes
+ * should be applied.
+ */
+ resolver = GetConflictResolver(subid, type, rel, *conflictslot,
+ NULL, &apply_remote);
+
ReportApplyConflict(estate, resultRelInfo, type, resolver,
NULL, *conflictslot, slot, uniqueidx,
xmin, origin, committs, apply_remote);
diff --git a/src/backend/replication/logical/applyparallelworker.c b/src/backend/replication/logical/applyparallelworker.c
index e7f7d4c..6cbfab0 100644
--- a/src/backend/replication/logical/applyparallelworker.c
+++ b/src/backend/replication/logical/applyparallelworker.c
@@ -161,6 +161,7 @@
#include "libpq/pqmq.h"
#include "pgstat.h"
#include "postmaster/interrupt.h"
+#include "replication/conflict.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
#include "replication/origin.h"
@@ -312,6 +313,18 @@ pa_can_start(void)
if (!AllTablesyncsReady())
return false;
+ /*
+ * Do not start a new parallel worker if 'last_update_wins' is configured
+ * for any conflict type, as we need the commit timestamp in the
+ * beginning.
+ *
+ * XXX: To lift this restriction, we could write the changes to a file
+ * when a conflict is detected, and then at the commit time, let the
+ * remaining changes be applied by the apply worker.
+ */
+ if (CheckIfSubHasTimeStampResolver(MySubscription->oid))
+ return false;
+
return true;
}
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 6bf9f0a..40dbfbf 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -34,6 +34,7 @@
static const char *const ConflictResolverNames[] = {
[CR_APPLY_REMOTE] = "apply_remote",
[CR_KEEP_LOCAL] = "keep_local",
+ [CR_LAST_UPDATE_WINS] = "last_update_wins",
[CR_APPLY_OR_SKIP] = "apply_or_skip",
[CR_APPLY_OR_ERROR] = "apply_or_error",
[CR_SKIP] = "skip",
@@ -76,19 +77,22 @@ static ConflictInfo ConflictInfoMap[] = {
.conflict_type = CT_INSERT_EXISTS,
.conflict_name = "insert_exists",
.default_resolver = CR_ERROR,
- .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true},
+ .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true,
+ [CR_LAST_UPDATE_WINS]=true},
},
[CT_UPDATE_EXISTS] = {
.conflict_type = CT_UPDATE_EXISTS,
.conflict_name = "update_exists",
.default_resolver = CR_ERROR,
- .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true},
+ .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true,
+ [CR_LAST_UPDATE_WINS]=true},
},
[CT_UPDATE_ORIGIN_DIFFERS] = {
.conflict_type = CT_UPDATE_ORIGIN_DIFFERS,
.conflict_name = "update_origin_differs",
.default_resolver = CR_APPLY_REMOTE,
- .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true},
+ .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true,
+ [CR_LAST_UPDATE_WINS]=true},
},
[CT_UPDATE_MISSING] = {
.conflict_type = CT_UPDATE_MISSING,
@@ -106,7 +110,8 @@ static ConflictInfo ConflictInfoMap[] = {
.conflict_type = CT_DELETE_ORIGIN_DIFFERS,
.conflict_name = "delete_origin_differs",
.default_resolver = CR_APPLY_REMOTE,
- .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true},
+ .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true,
+ [CR_LAST_UPDATE_WINS]=true},
},
};
@@ -411,6 +416,11 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (val_desc)
appendStringInfo(&err_detail, "\n%s", val_desc);
+ /* append remote tuple timestamp details if resolver is last_update_wins */
+ if (resolver == CR_LAST_UPDATE_WINS)
+ appendStringInfo(&err_detail, _(" The remote tuple is from orign-id=%u, modified at %s."),
+ replorigin_session_origin,
+ timestamptz_to_str(replorigin_session_origin_timestamp));
return errdetail_internal("%s", err_detail.data);
}
@@ -693,6 +703,14 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
conflict_resolver,
conflict_type));
+ if ((resolver == CR_LAST_UPDATE_WINS) && !track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ conflict_resolver, "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+
return type;
}
@@ -737,6 +755,42 @@ get_conflict_resolver_internal(ConflictType type, Oid subid)
}
/*
+ * Compare the timestamps of given local tuple and the remote tuple to
+ * resolve the conflict.
+ *
+ * Returns true if remote tuple has the latest timestamp, false otherwise.
+ */
+static bool
+resolve_by_timestamp(TupleTableSlot *localslot)
+{
+ TransactionId local_xmin;
+ TimestampTz local_ts;
+ RepOriginId local_origin;
+ int ts_cmp;
+ uint64 local_system_identifier;
+
+ /* Get origin and timestamp info of the local tuple */
+ GetTupleTransactionInfo(localslot, &local_xmin, &local_origin, &local_ts);
+
+ /* Compare the timestamps of remote & local tuple to decide the winner */
+ ts_cmp = timestamptz_cmp_internal(replorigin_session_origin_timestamp,
+ local_ts);
+
+ if (ts_cmp == 0)
+ {
+ elog(LOG, "Timestamps of remote and local tuple are equal, comparing remote and local system identifiers");
+
+ /* Get current system's identifier */
+ local_system_identifier = GetSystemIdentifier();
+
+ return local_system_identifier <= replorigin_session_origin_sysid;
+ }
+ else
+ return (ts_cmp > 0);
+
+}
+
+/*
* Check if a full tuple can be created from the new tuple.
* Return true if yes, false otherwise.
*/
@@ -775,7 +829,8 @@ can_create_full_tuple(Relation localrel,
*/
List *
ParseAndGetSubConflictResolvers(ParseState *pstate,
- List *stmtresolvers, bool add_defaults)
+ List *stmtresolvers, bool add_defaults,
+ bool sub_twophase)
{
List *res = NIL;
bool already_seen[CONFLICT_NUM_TYPES] = {0};
@@ -800,6 +855,23 @@ ParseAndGetSubConflictResolvers(ParseState *pstate,
already_seen[type] = true;
+ /*
+ * Time based conflict resolution for two phase transactions can
+ * result in data divergence, so disallow last_update_wins resolver
+ * when two_phase is enabled.
+ *
+ * XXX: An alternative solution idea is that if a conflict is detected
+ * and the resolution strategy is last_update_wins, then start writing
+ * all the changes to a file similar to what we do for streaming mode.
+ * Once commit_prepared arrives, we will read and apply the changes.
+ */
+ if ((pg_strcasecmp(resolver, "last_update_wins") == 0) &&
+ sub_twophase)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot set %s resolver when \"%s\" is enabled; these options are mutually exclusive",
+ "last_update_wins", "two_phase")));
+
conftyperesolver = palloc(sizeof(ConflictTypeResolver));
conftyperesolver->conflict_type_name = defel->defname;
conftyperesolver->conflict_resolver_name = resolver;
@@ -1037,14 +1109,30 @@ RemoveSubConflictResolvers(Oid subid)
*/
ConflictResolver
GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
- LogicalRepTupleData *newtup, bool *apply_remote)
+ TupleTableSlot *conflictslot, LogicalRepTupleData *newtup,
+ bool *apply_remote)
{
ConflictResolver resolver;
resolver = get_conflict_resolver_internal(type, subid);
+ /* If caller has given apply_remote as NULL, simply return the resolver */
+ if (!apply_remote)
+ return resolver;
+
switch (resolver)
{
+ case CR_LAST_UPDATE_WINS:
+ if (!track_commit_timestamp)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("resolver %s requires \"%s\" to be enabled",
+ ConflictResolverNames[resolver], "track_commit_timestamp"),
+ errhint("Make sure the configuration parameter \"%s\" is set.",
+ "track_commit_timestamp"));
+ else
+ *apply_remote = resolve_by_timestamp(conflictslot);
+ break;
case CR_APPLY_REMOTE:
*apply_remote = true;
break;
@@ -1072,3 +1160,40 @@ GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
return resolver;
}
+
+/*
+ * Check if the "last_update_wins" resolver is configured
+ * for any conflict type in the given subscription.
+ * Returns true if set, false otherwise.
+ */
+bool
+CheckIfSubHasTimeStampResolver(Oid subid)
+{
+ bool found = false;
+ bool started_tx = false;
+ ConflictType type;
+ ConflictResolver resolver;
+
+ /* This function might be called inside or outside of transaction */
+ if (!IsTransactionState())
+ {
+ StartTransactionCommand();
+ started_tx = true;
+ }
+
+ /* Check if any conflict type has resolver set to last_update_wins */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ resolver = get_conflict_resolver_internal(type, subid);
+ if (resolver == CR_LAST_UPDATE_WINS)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (started_tx)
+ CommitTransactionCommand();
+
+ return found;
+}
diff --git a/src/backend/replication/logical/origin.c b/src/backend/replication/logical/origin.c
index baf696d..3094030 100644
--- a/src/backend/replication/logical/origin.c
+++ b/src/backend/replication/logical/origin.c
@@ -159,6 +159,7 @@ typedef struct ReplicationStateCtl
RepOriginId replorigin_session_origin = InvalidRepOriginId; /* assumed identity */
XLogRecPtr replorigin_session_origin_lsn = InvalidXLogRecPtr;
TimestampTz replorigin_session_origin_timestamp = 0;
+uint64 replorigin_session_origin_sysid = 0;
/*
* Base address into a shared memory array of replication states of size
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3579bd0..291e79a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1006,6 +1006,12 @@ apply_handle_begin(StringInfo s)
in_remote_transaction = true;
pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Capture the commit timestamp of the remote transaction for time based
+ * conflict resolution purpose.
+ */
+ replorigin_session_origin_timestamp = begin_data.committime;
}
/*
@@ -2741,7 +2747,7 @@ apply_handle_update_internal(ApplyExecutionData *edata,
resolver = GetConflictResolver(MySubscription->oid,
CT_UPDATE_ORIGIN_DIFFERS,
- localrel, NULL,
+ localrel, localslot, NULL,
&apply_remote);
/* Store the new tuple for conflict reporting */
@@ -2834,7 +2840,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
* UPDATE to INSERT and apply the change.
*/
resolver = GetConflictResolver(MySubscription->oid, CT_UPDATE_MISSING,
- localrel, newtup, &apply_remote);
+ localrel, localslot, newtup,
+ &apply_remote);
ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
remoteslot, NULL, newslot, InvalidOid,
@@ -2989,7 +2996,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
{
resolver = GetConflictResolver(MySubscription->oid,
CT_DELETE_ORIGIN_DIFFERS,
- localrel, NULL, &apply_remote);
+ localrel, localslot, NULL,
+ &apply_remote);
ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
resolver, remoteslot, localslot, NULL,
@@ -3017,7 +3025,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
* configured, either skip and log a message or emit an error.
*/
resolver = GetConflictResolver(MySubscription->oid, CT_DELETE_MISSING,
- localrel, NULL, &apply_remote);
+ localrel, localslot, NULL,
+ &apply_remote);
/* Resolver is set to skip, thus report the conflict and skip */
if (!apply_remote)
@@ -3214,8 +3223,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
resolver = GetConflictResolver(MySubscription->oid,
- CT_UPDATE_MISSING,
- partrel, newtup,
+ CT_UPDATE_MISSING, partrel,
+ localslot, newtup,
&apply_remote);
/*
@@ -3259,7 +3268,7 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
resolver = GetConflictResolver(MySubscription->oid,
CT_UPDATE_ORIGIN_DIFFERS,
- partrel, NULL,
+ partrel, localslot, NULL,
&apply_remote);
ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
@@ -4790,6 +4799,7 @@ run_apply_worker()
TimeLineID startpointTLI;
char *err;
bool must_use_password;
+ char *replorigin_sysid;
slotname = MySubscription->slotname;
@@ -4830,10 +4840,12 @@ run_apply_worker()
MySubscription->name, err)));
/*
- * We don't really use the output identify_system for anything but it does
- * some initializations on the upstream so let's still call it.
+ * Call identify_system to do some initializations on the upstream and
+ * store the output as system identifier of the replication origin node.
*/
- (void) walrcv_identify_system(LogRepWorkerWalRcvConn, &startpointTLI);
+ replorigin_sysid = walrcv_identify_system(LogRepWorkerWalRcvConn,
+ &startpointTLI);
+ replorigin_session_origin_sysid = strtoul(replorigin_sysid, NULL, 10);
set_apply_error_context_origin(originname);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index 0680ef1..9eca7fc 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -9,6 +9,7 @@
#ifndef CONFLICT_H
#define CONFLICT_H
+#include "catalog/pg_subscription.h"
#include "nodes/execnodes.h"
#include "parser/parse_node.h"
#include "replication/logicalrelation.h"
@@ -66,6 +67,9 @@ typedef enum ConflictResolver
/* Keep the local change */
CR_KEEP_LOCAL,
+ /* Apply the change with latest timestamp */
+ CR_LAST_UPDATE_WINS,
+
/* Apply the remote change; skip if it cannot be applied */
CR_APPLY_OR_SKIP,
@@ -105,7 +109,8 @@ extern void SetSubConflictResolvers(Oid subId, List *resolvers);
extern void RemoveSubConflictResolvers(Oid confid);
extern List *ParseAndGetSubConflictResolvers(ParseState *pstate,
List *stmtresolvers,
- bool add_defaults);
+ bool add_defaults,
+ bool sub_twophase);
extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
extern ConflictType ValidateConflictType(const char *conflict_type);
extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
@@ -114,6 +119,8 @@ extern List *GetDefaultConflictResolvers(void);
extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
extern ConflictResolver GetConflictResolver(Oid subid, ConflictType type,
Relation localrel,
+ TupleTableSlot *localslot,
LogicalRepTupleData *newtup,
bool *apply_remote);
+extern bool CheckIfSubHasTimeStampResolver(Oid subid);
#endif
diff --git a/src/include/replication/origin.h b/src/include/replication/origin.h
index 7189ba9..dcbbbdf 100644
--- a/src/include/replication/origin.h
+++ b/src/include/replication/origin.h
@@ -36,6 +36,7 @@ typedef struct xl_replorigin_drop
extern PGDLLIMPORT RepOriginId replorigin_session_origin;
extern PGDLLIMPORT XLogRecPtr replorigin_session_origin_lsn;
extern PGDLLIMPORT TimestampTz replorigin_session_origin_timestamp;
+extern PGDLLIMPORT uint64 replorigin_session_origin_sysid;
/* API for querying & manipulating replication origins */
extern RepOriginId replorigin_by_name(const char *roname, bool missing_ok);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index f7c8abc..f6a7258 100644
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -156,6 +156,34 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+#############################################
+# Test 'last_update_wins' for 'insert_exists'
+#############################################
+
+# Change CONFLICT RESOLVER of insert_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=4);");
+
+is($result, 'frompub', "remote data wins");
+
# Truncate the table on the publisher
$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
@@ -256,6 +284,36 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+############################################
+# Test 'last_update_wins' for 'update_exists'
+############################################
+
+# Change CONFLICT RESOLVER of update_exists to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'last_update_wins');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'fromsub')");
+
+# Create conflicting data on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=5 WHERE a=3;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=5);");
+
+is($result, 'frompub', "remote data wins");
+
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -309,16 +367,48 @@ $node_subscriber->safe_psql(
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'delete_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'delete_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of delete_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
"INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
- INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (4,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=4);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=last_update_wins/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=4);");
+
+is($result, '', "delete from remote wins");
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'apply_remote');"
+);
# Modify data on the subscriber
$node_subscriber->safe_psql('postgres',
@@ -405,10 +495,14 @@ $node_subscriber->safe_psql(
# Wait for initial table sync to finish
$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
-#################################################
-# Test 'apply_remote' for 'update_origin_differs'
-#################################################
+#####################################################
+# Test 'last_update_wins' for 'update_origin_differs'
+#####################################################
+# Change CONFLICT RESOLVER of update_origin_differs to last_update_wins
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'last_update_wins');"
+);
# Insert data in the publisher
$node_publisher->safe_psql(
'postgres',
@@ -426,7 +520,7 @@ $node_publisher->safe_psql('postgres',
"UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
$node_subscriber->wait_for_log(
- qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=last_update_wins/,
$log_offset);
# Confirm that the remote update overrides the local update
@@ -435,6 +529,34 @@ $result = $node_subscriber->safe_psql('postgres',
is($result, 'frompubnew', "update from remote is kept");
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub2' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew2', "update from remote is kept");
+
###############################################
# Test 'keep_local' for 'update_origin_differs'
###############################################
--
1.8.3.1
v17-0003-Conflict-resolution-for-update_exists-conflict-t.patchapplication/octet-stream; name=v17-0003-Conflict-resolution-for-update_exists-conflict-t.patchDownload
From 5e4e738f0e50624917bfe1efebb9c51b58c11e01 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 18 Oct 2024 03:20:47 -0400
Subject: [PATCH v17 3/4] Conflict resolution for update_exists conflict type.
Supports following resolutions:
- 'apply_remote': Deletes the conflicting local tuple and inserts the remote tuple.
- 'keep_local': Keeps the local tuple and ignores the remote one.
- 'error': Triggers an error.
---
src/backend/executor/execReplication.c | 56 +++++--
src/backend/replication/logical/worker.c | 104 ++++++++++--
src/include/executor/executor.h | 3 +-
src/test/subscription/t/034_conflict_resolver.pl | 201 +++++++++++++++++++++++
4 files changed, 339 insertions(+), 25 deletions(-)
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 27d6e98..cb6021b 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -486,7 +486,8 @@ retry:
static bool
FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
Oid conflictindex, TupleTableSlot *slot,
- TupleTableSlot **conflictslot)
+ TupleTableSlot **conflictslot, ItemPointer tupleid,
+ LockTupleMode lockmode)
{
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointerData conflictTid;
@@ -497,7 +498,7 @@ FindConflictTuple(ResultRelInfo *resultRelInfo, EState *estate,
retry:
if (ExecCheckIndexConstraints(resultRelInfo, slot, estate,
- &conflictTid, &slot->tts_tid,
+ &conflictTid, tupleid,
list_make1_oid(conflictindex)))
{
if (*conflictslot)
@@ -514,7 +515,7 @@ retry:
res = table_tuple_lock(rel, &conflictTid, GetLatestSnapshot(),
*conflictslot,
GetCurrentCommandId(false),
- LockTupleShare,
+ lockmode,
LockWaitBlock,
0 /* don't follow updates */ ,
&tmfd);
@@ -543,7 +544,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
if (list_member_oid(recheckIndexes, uniqueidx) &&
FindConflictTuple(resultRelInfo, estate, uniqueidx, remoteslot,
- &conflictslot))
+ &conflictslot, &remoteslot->tts_tid,
+ LockTupleShare))
{
RepOriginId origin;
TimestampTz committs;
@@ -564,17 +566,21 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
* tuple information in conflictslot.
*/
static bool
-has_conflicting_tuple(Oid subid, ConflictType type,
+has_conflicting_tuple(Oid subid, ConflictType type, ItemPointer tupleid,
ResultRelInfo *resultRelInfo, EState *estate,
TupleTableSlot *slot, TupleTableSlot **conflictslot)
{
ConflictResolver resolver;
+ LockTupleMode lockmode;
bool apply_remote = false;
Relation rel = resultRelInfo->ri_RelationDesc;
List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
- /* ASSERT if called for any conflict type other than insert_exists */
- Assert(type == CT_INSERT_EXISTS);
+ /*
+ * ASSERT if called for any conflict type other than insert_exists or
+ * update_exists
+ */
+ Assert(type == CT_INSERT_EXISTS || type == CT_UPDATE_EXISTS);
/*
* Get the configured resolver and determine if remote changes should be
@@ -583,6 +589,17 @@ has_conflicting_tuple(Oid subid, ConflictType type,
resolver = GetConflictResolver(subid, type, rel, NULL, &apply_remote);
/*
+ * Determine the lock mode for the conflicting tuple, if any. Take an
+ * exclusive lock if the resolution favors modifying the conflicting
+ * tuple; otherwise, take a shared lock.
+ */
+
+ if (apply_remote)
+ lockmode = LockTupleExclusive;
+ else
+ lockmode = LockTupleShare;
+
+ /*
* Proceed to find conflict if the resolver is set to a non-default value;
* if the resolver is 'ERROR' (default), the caller will handle it.
*/
@@ -594,7 +611,7 @@ has_conflicting_tuple(Oid subid, ConflictType type,
{
/* Return to caller for resolutions if any conflict is found */
if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
- &(*conflictslot)))
+ &(*conflictslot), tupleid, lockmode))
{
RepOriginId origin;
TimestampTz committs;
@@ -651,6 +668,7 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
List *recheckIndexes = NIL;
List *conflictindexes;
bool conflict = false;
+ ItemPointerData invalidItemPtr;
/* Compute stored generated columns */
if (rel->rd_att->constr &&
@@ -673,8 +691,9 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
* inserted tuple if a conflict is detected after insertion with a
* non-default resolution set.
*/
- if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, resultRelInfo,
- estate, slot, &(*conflictslot)))
+ if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, &invalidItemPtr,
+ resultRelInfo, estate, slot,
+ &(*conflictslot)))
return;
/* OK, store the tuple and create index entries for it */
@@ -732,7 +751,8 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
void
ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot)
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -775,6 +795,20 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Check for conflict and return to caller for resolution, if found.
+ *
+ * XXX In case there are no conflicts, a non-default 'update_exists'
+ * resolver adds overhead by performing an extra scan here. However,
+ * this approach avoids the extra work needed to rollback/delete the
+ * updated tuple if a conflict is detected after update with a
+ * non-default resolution set.
+ */
+ if (has_conflicting_tuple(subid, CT_UPDATE_EXISTS, tid,
+ resultRelInfo, estate, slot,
+ &(*conflictslot)))
+ return;
+
simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,
&update_indexes);
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index abda71a..3579bd0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -389,7 +389,8 @@ static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid);
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot);
static void apply_handle_delete_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2518,8 +2519,9 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the update */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
- remoteslot);
+ apply_handle_update_internal(edata, relinfo, remoteslot, newtup,
+ rel_entry->localindexoid, true,
+ conflictslot);
EvalPlanQualEnd(&epqstate);
ExecDropSingleTupleTableSlot(conflictslot);
@@ -2671,7 +2673,8 @@ apply_handle_update(StringInfo s)
remoteslot, &newtup, CMD_UPDATE);
else
apply_handle_update_internal(edata, edata->targetRelInfo,
- remoteslot, &newtup, rel->localindexoid);
+ remoteslot, &newtup, rel->localindexoid,
+ false, NULL);
finish_edata(edata);
@@ -2696,14 +2699,13 @@ apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
LogicalRepTupleData *newtup,
- Oid localindexoid)
+ Oid localindexoid, bool found,
+ TupleTableSlot *localslot)
{
EState *estate = edata->estate;
LogicalRepRelMapEntry *relmapentry = edata->targetRel;
Relation localrel = relinfo->ri_RelationDesc;
EPQState epqstate;
- TupleTableSlot *localslot;
- bool found;
MemoryContext oldctx;
bool apply_remote = true;
ConflictResolver resolver;
@@ -2711,10 +2713,11 @@ apply_handle_update_internal(ApplyExecutionData *edata,
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
- found = FindReplTupleInLocalRel(edata, localrel,
- &relmapentry->remoterel,
- localindexoid,
- remoteslot, &localslot);
+ if (!found)
+ found = FindReplTupleInLocalRel(edata, localrel,
+ &relmapentry->remoterel,
+ localindexoid,
+ remoteslot, &localslot);
/*
* Tuple found.
@@ -2756,6 +2759,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
*/
if (apply_remote)
{
+ TupleTableSlot *conflictslot = NULL;
+
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
slot_modify_data(remoteslot, localslot, relmapentry, newtup);
@@ -2768,7 +2773,52 @@ apply_handle_update_internal(ApplyExecutionData *edata,
/* Do the actual update. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete the found
+ * tuple and apply the remote update to the conflicting tuple.
+ *
+ * If the local table contains multiple unique constraint columns
+ * and the conflict resolution strategy favors applying the remote
+ * changes, then a remote INSERT or UPDATE could trigger multiple
+ * update_exists conflicts. Each conflict will be detected and
+ * resolved in sequence (recursively), possibly resulting in the
+ * deletion of multiple local rows.
+ *
+ * Consider the scenario: A table where all columns have unique
+ * indexes, and the first column is the replica identity. The
+ * publisher has (1, 1, 1) while the subscriber has three rows:
+ * (1, 1, 1), (2, 2, 2), and (3, 3, 3). The publisher updates (1,
+ * 1, 1) to (1, 2, 3).
+ *
+ * TThe current logic works as follows: We find the row (1, 1, 1)
+ * and try to update it to (1, 2, 3), but find a conflict with the
+ * tuple (2, 2, 2). We delete (1, 1, 1), and then try to update
+ * (2, 2, 2) to (1, 2, 3), but encounter another conflict with (3,
+ * 3, 3). We then delete (2, 2, 2) and try to update (3, 3, 3) to
+ * (1, 2, 3).
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, relinfo,
+ remoteslot, newtup,
+ relmapentry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
}
}
else
@@ -3269,6 +3319,8 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
* conflict is detected for the found tuple and the
* resolver is in favour of applying the update.
*/
+ TupleTableSlot *conflictslot = NULL;
+
ExecOpenIndices(partrelinfo, true);
InitConflictIndexes(partrelinfo);
@@ -3276,7 +3328,33 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
ACL_UPDATE);
ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ localslot, remoteslot_part,
+ &conflictslot, MySubscription->oid);
+
+ /*
+ * If an update_exists conflict is detected, delete
+ * the found tuple and apply the remote update to the
+ * conflicting tuple.
+ */
+ if (conflictslot)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, conflictslot, part_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ /* Delete the found row */
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+
+ /* Update the conflicting tuple with remote update */
+ apply_handle_update_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry->localindexoid,
+ true, conflictslot);
+
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
+
ExecCloseIndices(partrelinfo);
}
else if (apply_remote)
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index f390975..291d5dd 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -673,7 +673,8 @@ extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
- TupleTableSlot *searchslot, TupleTableSlot *slot);
+ TupleTableSlot *searchslot, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid);
extern void ExecSimpleRelationDelete(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot);
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
index 86e3ad1..f7c8abc 100644
--- a/src/test/subscription/t/034_conflict_resolver.pl
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -156,6 +156,106 @@ $node_subscriber->wait_for_log(
# Truncate table on subscriber to get rid of the error
$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'fromsub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (4,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (5,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (6,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres', "UPDATE conf_tab SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_exists, resolution=error/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
###################################
# Test 'skip' for 'delete_missing'
###################################
@@ -874,4 +974,105 @@ $node_subscriber->wait_for_log(
qr/ERROR: conflict detected on relation \"public.conf_tab_part_2\": conflict=update_missing, resolution=error/,
$log_offset);
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+############################################
+# Test 'apply_remote' for 'update_exists'
+############################################
+# Change CONFLICT RESOLVER of update_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'apply_remote');"
+);
+
+# Insert data in the subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (2,1,'fromsub');
+ INSERT INTO conf_tab_part VALUES (3,1,'fromsub');");
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (4,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (5,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (6,1,'frompub');");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=1 WHERE a=4;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote update is applied.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=1);");
+
+is($result, 'frompub', "update from remote on partition is kept");
+
+########################################
+# Test 'keep_local' for 'update_exists'
+########################################
+
+# Change CONFLICT RESOLVER of update_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'keep_local');"
+);
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=2 WHERE a=5;");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from remote on partition is skipped");
+
+###################################
+# Test 'error' for 'update_exists'
+###################################
+
+# Change CONFLICT RESOLVER of update_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_exists = 'error');"
+);
+
+# Update on publisher which already exists on subscriber
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET a=3 WHERE a=6;");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_exists, resolution=error/,
+ $log_offset);
+
done_testing();
--
1.8.3.1
v17-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchapplication/octet-stream; name=v17-0001-Add-CONFLICT-RESOLVERS-into-the-syntax-for-CREAT.patchDownload
From e02b058f2e932a3f7e8c80aea2f63b59e419366f Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Mon, 14 Oct 2024 05:35:35 -0400
Subject: [PATCH v17 1/4] Add CONFLICT RESOLVERS into the syntax for CREATE and
ALTER SUBSCRIPTION
This patch provides support for configuring subscriptions with conflict resolvers.
Syntax for CREATE SUBSCRIPTION:
CREATE SUBSCRIPTION <subname> CONNECTION <conninfo> PUBLICATION <pubname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for ALTER SUBSCRIPTION:
ALTER SUBSCRIPTION <subname> CONFLICT RESOLVER
(conflict_type1 = resolver1, conflict_type2 = resolver2, conflict_type3 = resolver3,...);
Syntax for resetting conflict resolvers:
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER ALL
ALTER SUBSCRIPTION <subname> RESET CONFLICT RESOLVER FOR conflict_type
Additionally, this patch includes documentation for conflict resolvers.
---
doc/src/sgml/logical-replication.sgml | 95 +-----
doc/src/sgml/monitoring.sgml | 13 +-
doc/src/sgml/ref/alter_subscription.sgml | 63 ++++
doc/src/sgml/ref/create_subscription.sgml | 211 ++++++++++++
src/backend/commands/subscriptioncmds.c | 53 +++
src/backend/parser/gram.y | 49 ++-
src/backend/replication/logical/conflict.c | 451 ++++++++++++++++++++++++-
src/bin/pg_dump/pg_dump.c | 126 ++++++-
src/bin/pg_dump/pg_dump.h | 8 +
src/bin/pg_dump/t/002_pg_dump.pl | 3 +-
src/bin/psql/describe.c | 118 ++++++-
src/bin/psql/tab-complete.in.c | 22 +-
src/include/catalog/Makefile | 3 +-
src/include/catalog/meson.build | 1 +
src/include/catalog/pg_subscription_conflict.h | 55 +++
src/include/nodes/parsenodes.h | 7 +
src/include/parser/kwlist.h | 1 +
src/include/replication/conflict.h | 49 ++-
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/subscription.out | 362 ++++++++++++++++++--
src/test/regress/sql/subscription.sql | 63 ++++
src/tools/pgindent/typedefs.list | 3 +
22 files changed, 1627 insertions(+), 130 deletions(-)
create mode 100644 src/include/catalog/pg_subscription_conflict.h
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 98a7ad0..86c9b0b 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -1568,7 +1568,7 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</sect1>
<sect1 id="logical-replication-conflicts">
- <title>Conflicts</title>
+ <title>Conflicts and Conflict Resolution</title>
<para>
Logical replication behaves similarly to normal DML operations in that
@@ -1582,86 +1582,19 @@ test_sub=# SELECT * FROM t1 ORDER BY id;
</para>
<para>
- Additional logging is triggered, and the conflict statistics are collected (displayed in the
- <link linkend="monitoring-pg-stat-subscription-stats"><structname>pg_stat_subscription_stats</structname></link> view)
- in the following <firstterm>conflict</firstterm> cases:
- <variablelist>
- <varlistentry id="conflict-insert-exists" xreflabel="insert_exists">
- <term><literal>insert_exists</literal></term>
- <listitem>
- <para>
- Inserting a row that violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-origin-differs" xreflabel="update_origin_differs">
- <term><literal>update_origin_differs</literal></term>
- <listitem>
- <para>
- Updating a row that was previously modified by another origin.
- Note that this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the update is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-exists" xreflabel="update_exists">
- <term><literal>update_exists</literal></term>
- <listitem>
- <para>
- The updated value of a row violates a <literal>NOT DEFERRABLE</literal>
- unique constraint. Note that to log the origin and commit
- timestamp details of the conflicting key,
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- should be enabled on the subscriber. In this case, an error will be
- raised until the conflict is resolved manually. Note that when updating a
- partitioned table, if the updated row value satisfies another partition
- constraint resulting in the row being inserted into a new partition, the
- <literal>insert_exists</literal> conflict may arise if the new row
- violates a <literal>NOT DEFERRABLE</literal> unique constraint.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-update-missing" xreflabel="update_missing">
- <term><literal>update_missing</literal></term>
- <listitem>
- <para>
- The tuple to be updated was not found. The update will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-origin-differs" xreflabel="delete_origin_differs">
- <term><literal>delete_origin_differs</literal></term>
- <listitem>
- <para>
- Deleting a row that was previously modified by another origin. Note that
- this conflict can only be detected when
- <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
- is enabled on the subscriber. Currently, the delete is always applied
- regardless of the origin of the local row.
- </para>
- </listitem>
- </varlistentry>
- <varlistentry id="conflict-delete-missing" xreflabel="delete_missing">
- <term><literal>delete_missing</literal></term>
- <listitem>
- <para>
- The tuple to be deleted was not found. The delete will simply be
- skipped in this scenario.
- </para>
- </listitem>
- </varlistentry>
- </variablelist>
- Note that there are other conflict scenarios, such as exclusion constraint
- violations. Currently, we do not provide additional details for them in the
- log.
+ There are various conflict scenarios, each identified as a <firstterm>conflict type</firstterm>.
+ Users can configure a <firstterm>conflict resolver</firstterm> for each
+ conflict type when creating a subscription. For more information, refer to
+ <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIPTION ... CONFLICT RESOLVER</command></link>.
+ </para>
+ <para>
+ When a conflict occurs the details about it are logged, and the conflict
+ statistics are recorded in the <link linkend="monitoring-pg-stat-subscription-stats">
+ <structname>pg_stat_subscription_stats</structname></link> view.
+ Note that there are other conflict scenarios, such as exclusion constraint
+ violations. Currently, we do not provide additional details for them in the
+ log.
</para>
<para>
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 331315f..d79db76 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2181,7 +2181,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a row insertion violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-insert-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-insert-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2193,7 +2193,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times an update was applied to a row that had been previously
modified by another source during the application of changes. See
- <xref linkend="conflict-update-origin-differs"/> for details about this
+ <xref linkend="sql-createsubscription-params-with-conflict_type-update-origin-differs"/> for details about this
conflict.
</para></entry>
</row>
@@ -2205,7 +2205,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times that an updated row value violated a
<literal>NOT DEFERRABLE</literal> unique constraint during the
- application of changes. See <xref linkend="conflict-update-exists"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-exists"/>
for details about this conflict.
</para></entry>
</row>
@@ -2216,7 +2216,8 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be updated was not found during the
- application of changes. See <xref linkend="conflict-update-missing"/>
+ application of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-update-missing
+"/>
for details about this conflict.
</para></entry>
</row>
@@ -2228,7 +2229,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
<para>
Number of times a delete operation was applied to row that had been
previously modified by another source during the application of changes.
- See <xref linkend="conflict-delete-origin-differs"/> for details about
+ See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-origin-differs"/> for details about
this conflict.
</para></entry>
</row>
@@ -2239,7 +2240,7 @@ description | Waiting for a newly initialized WAL file to reach durable storage
</para>
<para>
Number of times the tuple to be deleted was not found during the application
- of changes. See <xref linkend="conflict-delete-missing"/> for details
+ of changes. See <xref linkend="sql-createsubscription-params-with-conflict_type-delete-missing"/> for details
about this conflict.
</para></entry>
</row>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index fdc648d..8ed0b93 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -32,6 +32,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SET ( <repl
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> SKIP ( <replaceable class="parameter">skip_option</replaceable> = <replaceable class="parameter">value</replaceable> )
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <replaceable>new_name</replaceable>
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] )
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER ALL
+ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RESET CONFLICT RESOLVER FOR '<replaceable class="parameter">conflict_type</replaceable>'
</synopsis>
</refsynopsisdiv>
@@ -345,6 +348,66 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
</para>
</listitem>
</varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This clause alters the current conflict resolver for the specified conflict types.
+ Refer to <link linkend="sql-createsubscription-params-with-conflict-resolver">
+ <command>CREATE SUBSCRIBER ... CONFLICT RESOLVER</command></link>
+ for details about different conflict types and what kinds of resolver can
+ be assigned to them.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-type">
+ <term><replaceable class="parameter">conflict_type</replaceable></term>
+ <listitem>
+ <para>
+ The conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-conflict-resolver-resolver">
+ <term><replaceable class="parameter">conflict_resolver</replaceable></term>
+ <listitem>
+ <para>
+ The conflict resolver to use for this conflict type.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-altersubscription-params-reset-conflict-resolver">
+ <term><literal>RESET CONFLICT RESOLVER</literal></term>
+ <listitem>
+ <para>
+ Reset all conflict types, or a specified conflict type, to their default resolvers.
+ For details on conflict types and their default resolvers, refer to
+ section <link linkend="sql-createsubscription-params-with-conflict-resolver"><literal>CREATE SUBSCRIBER ... CONFLICT RESOLVER</literal></link>.
+ </para>
+ <variablelist>
+ <varlistentry id="sql-altersubscription-params-reset-all">
+ <term><literal>ALL</literal></term>
+ <listitem>
+ <para>
+ All conflict types will be reset to their respective default resolvers.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry id="sql-altersubscription-params-reset-conflict-type">
+ <term><literal>FOR</literal> '<replaceable class="parameter">conflict_type</replaceable>'</term>
+ <listitem>
+ <para>
+ The given <replaceable class="parameter">conflict_type</replaceable> will be reset to its default resolver.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
</variablelist>
<para>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 8a3096e..06009c3 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -24,6 +24,7 @@ PostgreSQL documentation
CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceable>
CONNECTION '<replaceable class="parameter">conninfo</replaceable>'
PUBLICATION <replaceable class="parameter">publication_name</replaceable> [, ...]
+ [ CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ...] ) ]
[ WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
</synopsis>
</refsynopsisdiv>
@@ -97,6 +98,216 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry id="sql-createsubscription-params-with-conflict-resolver">
+ <term><literal>CONFLICT RESOLVER ( <replaceable class="parameter">conflict_type</replaceable> = <replaceable class="parameter">conflict_resolver</replaceable> [, ... ] )</literal></term>
+ <listitem>
+ <para>
+ This optional clause specifies conflict resolvers for different conflict types.
+ </para>
+
+ <table id="sql-createsubscription-params-conflict-type-resolver-summary">
+ <title>Conflict type/resolver Summary</title>
+ <tgroup cols="3">
+ <thead>
+ <row><entry>Conflict type</entry> <entry>Default resolver</entry> <entry>Possible resolvers</entry></row>
+ </thead>
+ <tbody>
+ <row><entry>insert_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_exists</entry> <entry>error</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>update_missing</entry> <entry>skip</entry> <entry>apply_or_error, apply_or_skip, error, skip</entry></row>
+ <row><entry>update_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ <row><entry>delete_missing</entry> <entry>skip</entry> <entry>error, skip</entry></row>
+ <row><entry>delete_origin_differs</entry><entry>apply_remote</entry> <entry>apply_remote, error, keep_local</entry></row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ The scenarios that trigger conflicts, along with the behavior for each
+ <replaceable class="parameter">conflict_type</replaceable>, are listed below.
+ <variablelist>
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-insert-exists" xreflabel="insert_exists">
+ <term><literal>insert_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when inserting a row that violates a
+ <literal>NOT DEFERRABLE</literal> unique constraint.
+ The default behavior is an error will be raised until the conflict
+ is resolved manually or the resolver is configured to a non-default
+ value that can automatically resolve the conflict.
+ </para>
+ <para>
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-exists" xreflabel="update_exists">
+ <term><literal>update_exists</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the updated value of a row violates
+ a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ The default behavior is an error will be raised until the conflict
+ is resolved manually or the resolver is configured to a non-default
+ value that can automatically resolve the conflict.
+ </para>
+ <para>
+ Note that when updating a partitioned table, if the updated row
+ value satisfies another partition constraint resulting in the
+ row being inserted into a new partition, the <literal>insert_exists</literal>
+ conflict may arise if the new row violates a <literal>NOT DEFERRABLE</literal>
+ unique constraint.
+ </para>
+ <para>
+ To log the origin and commit timestamp details of the conflicting key,
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ should be enabled on the subscriber.
+ </para>
+ <warning>
+ <para>
+ If the local table contains multiple <literal>NOT DEFERRABLE</literal> unique constraint
+ columns and the conflict resolution strategy for an INSERT or UPDATE favors applying the changes,
+ then a remote INSERT or UPDATE could trigger multiple <literal>update_exists</literal> conflicts.
+ Each conflict will be detected and resolved in sequence, potentially leading to the deletion of
+ multiple local rows.
+ </para>
+ </warning>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-missing" xreflabel="update_missing">
+ <term><literal>update_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be updated was not found.
+ The default behavior is to simply skip the update.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-update-origin-differs" xreflabel="update_origin_differs">
+ <term><literal>update_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when updating a row that was previously
+ modified by another origin.
+ The default behavoir is that the update will be applied regardless
+ of the origin of the local row.
+ </para>
+ <para>
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-missing" xreflabel="delete_missing">
+ <term><literal>delete_missing</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when the tuple to be deleted was not found.
+ The default behavior is to simply skip the delete.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_type-delete-origin-differs" xreflabel="delete_origin_differs">
+ <term><literal>delete_origin_differs</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This conflict occurs when deleting a row that was previously modified
+ by another origin.
+ The default behavior is that the delete will be applied regardless
+ of the origin of the local row.
+ </para>
+ <para>
+ Note that this conflict can only be detected when
+ <link linkend="guc-track-commit-timestamp"><varname>track_commit_timestamp</varname></link>
+ is enabled on the subscriber.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para>
+ The behavior of each <replaceable class="parameter">conflict_resolver</replaceable>
+ is described below. Users can choose from the following resolvers for automatic conflict
+ resolution.
+ <variablelist>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-error">
+ <term><literal>error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver throws an error and stops replication.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-skip">
+ <term><literal>skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver skips processing the remote change and continues replication
+ with the next change.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-remote">
+ <term><literal>apply_remote</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver applies the remote change.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-keep-local">
+ <term><literal>keep_local</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver maintains the local tuple; the remote change is not applied.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-error">
+ <term><literal>apply_or_error</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this cannot
+ be done due to missing information, then an error is thrown, and replication
+ is stopped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="sql-createsubscription-params-with-conflict_resolver-apply-or-skip">
+ <term><literal>apply_or_skip</literal> (<type>enum</type>)</term>
+ <listitem>
+ <para>
+ This resolver is only used for <literal>update_missing</literal>.
+ An attempt is made to convert the update into an insert; if this
+ cannot be done due to missing information, then the change is skipped.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </listitem>
+ </varlistentry>
+
<varlistentry id="sql-createsubscription-params-with">
<term><literal>WITH ( <replaceable class="parameter">subscription_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
<listitem>
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 02ccc63..54d8ad9 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -36,6 +36,7 @@
#include "executor/executor.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/logicallauncher.h"
#include "replication/logicalworker.h"
@@ -583,6 +584,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
bits32 supported_opts;
SubOpts opts = {0};
AclResult aclresult;
+ List *conflict_resolvers = NIL;
/*
* Parse and check options.
@@ -597,6 +599,10 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_ORIGIN);
parse_subscription_options(pstate, stmt->options, supported_opts, &opts);
+ /* Parse and get conflict resolvers list. */
+ conflict_resolvers =
+ ParseAndGetSubConflictResolvers(pstate, stmt->resolvers, true);
+
/*
* Since creating a replication slot is not transactional, rolling back
* the transaction leaves the created replication slot. So we cannot run
@@ -723,6 +729,9 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt,
recordDependencyOnOwner(SubscriptionRelationId, subid, owner);
+ /* Update the Conflict Resolvers in pg_subscription_conflict */
+ SetSubConflictResolvers(subid, conflict_resolvers);
+
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_create(originname);
@@ -1581,6 +1590,47 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt,
update_tuple = true;
break;
}
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Get the list of conflict types and resolvers and validate
+ * them.
+ */
+ conflict_resolvers = ParseAndGetSubConflictResolvers(
+ pstate,
+ stmt->resolvers,
+ false);
+
+ /*
+ * Update the conflict resolvers for the corresponding
+ * conflict types in the pg_subscription_conflict catalog.
+ */
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL:
+ {
+ List *conflict_resolvers = NIL;
+
+ /*
+ * Create list of conflict resolvers and set them in the
+ * catalog.
+ */
+ conflict_resolvers = GetDefaultConflictResolvers();
+ UpdateSubConflictResolvers(conflict_resolvers, subid);
+ break;
+ }
+ case ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET:
+ {
+ /*
+ * Reset the conflict resolver for this conflict type to its
+ * default.
+ */
+ ResetSubConflictResolver(subid, stmt->conflict_type_name);
+ break;
+ }
default:
elog(ERROR, "unrecognized ALTER SUBSCRIPTION kind %d",
@@ -1832,6 +1882,9 @@ DropSubscription(DropSubscriptionStmt *stmt, bool isTopLevel)
/* Remove any associated relation synchronization states. */
RemoveSubscriptionRel(subid, InvalidOid);
+ /* Remove any associated conflict resolvers */
+ RemoveSubConflictResolvers(subid);
+
/* Remove the origin tracking if exists. */
ReplicationOriginNameForLogicalRep(subid, InvalidOid, originname, sizeof(originname));
replorigin_drop_by_name(originname, true, false);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 4aa8646..28591c9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -425,7 +425,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
OptTableElementList TableElementList OptInherit definition
OptTypedTableElementList TypedTableElementList
reloptions opt_reloptions
- OptWith opt_definition func_args func_args_list
+ OptWith opt_definition opt_resolver_definition func_args func_args_list
func_args_with_defaults func_args_with_defaults_list
aggr_args aggr_args_list
func_as createfunc_opt_list opt_createfunc_opt_list alterfunc_opt_list
@@ -613,6 +613,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <ival> opt_check_option
%type <str> opt_provider security_label
+%type <str> conflict_type
%type <target> xml_attribute_el
%type <list> xml_attribute_list xml_attributes
@@ -770,7 +771,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RANGE READ REAL REASSIGN RECURSIVE REF_P REFERENCES REFERENCING
REFRESH REINDEX RELATIVE_P RELEASE RENAME REPEATABLE REPLACE REPLICA
- RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
+ RESET RESOLVER RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
@@ -8813,6 +8814,11 @@ opt_definition:
| /*EMPTY*/ { $$ = NIL; }
;
+opt_resolver_definition:
+ CONFLICT RESOLVER definition { $$ = $3; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
table_func_column: param_name func_type
{
FunctionParameter *n = makeNode(FunctionParameter);
@@ -10747,14 +10753,15 @@ AlterPublicationStmt:
*****************************************************************************/
CreateSubscriptionStmt:
- CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_definition
+ CREATE SUBSCRIPTION name CONNECTION Sconst PUBLICATION name_list opt_resolver_definition opt_definition
{
CreateSubscriptionStmt *n =
makeNode(CreateSubscriptionStmt);
n->subname = $3;
n->conninfo = $5;
n->publication = $7;
- n->options = $8;
+ n->resolvers = $8;
+ n->options = $9;
$$ = (Node *) n;
}
;
@@ -10861,6 +10868,38 @@ AlterSubscriptionStmt:
n->options = $5;
$$ = (Node *) n;
}
+ | ALTER SUBSCRIPTION name opt_resolver_definition
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS;
+ n->subname = $3;
+ n->resolvers = $4;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER ALL
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL;
+ n->subname = $3;
+ $$ = (Node *) n;
+ }
+ | ALTER SUBSCRIPTION name RESET CONFLICT RESOLVER FOR conflict_type
+ {
+ AlterSubscriptionStmt *n =
+ makeNode(AlterSubscriptionStmt);
+
+ n->kind = ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET;
+ n->subname = $3;
+ n->conflict_type_name = $8;
+ $$ = (Node *) n;
+ }
+ ;
+conflict_type:
+ Sconst { $$ = $1; }
;
/*****************************************************************************
@@ -17785,6 +17824,7 @@ unreserved_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
@@ -18414,6 +18454,7 @@ bare_label_keyword:
| REPLACE
| REPLICA
| RESET
+ | RESOLVER
| RESTART
| RESTRICT
| RETURN
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index 5d9ff62..cb6d50c 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -8,30 +8,111 @@
* src/backend/replication/logical/conflict.c
*
* This file contains the code for logging conflicts on the subscriber during
- * logical replication.
+ * logical replication and setting up conflict resolvers for a subscription.
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "access/commit_ts.h"
+#include "access/heapam.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_subscription_conflict_d.h"
+#include "commands/defrem.h"
#include "access/tableam.h"
#include "executor/executor.h"
+#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+
+static const char *const ConflictResolverNames[] = {
+ [CR_APPLY_REMOTE] = "apply_remote",
+ [CR_KEEP_LOCAL] = "keep_local",
+ [CR_APPLY_OR_SKIP] = "apply_or_skip",
+ [CR_APPLY_OR_ERROR] = "apply_or_error",
+ [CR_SKIP] = "skip",
+ [CR_ERROR] = "error"
+};
+
+StaticAssertDecl(lengthof(ConflictResolverNames) == CONFLICT_NUM_RESOLVERS,
+ "array length mismatch");
-static const char *const ConflictTypeNames[] = {
- [CT_INSERT_EXISTS] = "insert_exists",
- [CT_UPDATE_ORIGIN_DIFFERS] = "update_origin_differs",
- [CT_UPDATE_EXISTS] = "update_exists",
- [CT_UPDATE_MISSING] = "update_missing",
- [CT_DELETE_ORIGIN_DIFFERS] = "delete_origin_differs",
- [CT_DELETE_MISSING] = "delete_missing"
+/*
+ * Valid conflict resolvers for each conflict type.
+ *
+ * First member represents default resolver for each conflict_type.
+ * The same defaults are used in pg_dump.c. If any default is changed here,
+ * ensure the corresponding value is updated in pg_dump's is_default_resolver
+ * function.
+ *
+ * XXX: If we do not want to maintain different resolvers such as
+ * apply_or_skip and apply_or_error for update_missing conflict,
+ * then we can retain apply_remote and keep_local only. Then these
+ * resolvers in context of update_missing will mean:
+ *
+ * keep_local: do not apply the update as INSERT.
+ * apply_remote: apply the update as INSERT. If we could not apply,
+ * then log and skip.
+ *
+ * Similarly SKIP can be replaced with KEEP_LOCAL for both update_missing
+ * and delete_missing conflicts. For missing rows, 'SKIP' sounds more user
+ * friendly name for a resolver and thus has been added here.
+ */
+typedef struct ConflictInfo {
+ ConflictType conflict_type;
+ const char *conflict_name;
+ ConflictResolver default_resolver;
+ bool allowed_resolvers[CONFLICT_NUM_RESOLVERS];
+} ConflictInfo;
+
+static ConflictInfo ConflictInfoMap[] = {
+ [CT_INSERT_EXISTS] = {
+ .conflict_type = CT_INSERT_EXISTS,
+ .conflict_name = "insert_exists",
+ .default_resolver = CR_ERROR,
+ .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true},
+ },
+ [CT_UPDATE_EXISTS] = {
+ .conflict_type = CT_UPDATE_EXISTS,
+ .conflict_name = "update_exists",
+ .default_resolver = CR_ERROR,
+ .allowed_resolvers = {[CR_ERROR]=true, [CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true},
+ },
+ [CT_UPDATE_ORIGIN_DIFFERS] = {
+ .conflict_type = CT_UPDATE_ORIGIN_DIFFERS,
+ .conflict_name = "update_origin_differs",
+ .default_resolver = CR_APPLY_REMOTE,
+ .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true},
+ },
+ [CT_UPDATE_MISSING] = {
+ .conflict_type = CT_UPDATE_MISSING,
+ .conflict_name = "update_missing",
+ .default_resolver = CR_SKIP,
+ .allowed_resolvers = {[CR_SKIP]=true, [CR_APPLY_OR_SKIP]=true, [CR_APPLY_OR_ERROR]=true, [CR_ERROR]=true},
+ },
+ [CT_DELETE_MISSING] = {
+ .conflict_type = CT_DELETE_MISSING,
+ .conflict_name = "delete_missing",
+ .default_resolver = CR_SKIP,
+ .allowed_resolvers = {[CR_SKIP]=true, [CR_ERROR]=true},
+ },
+ [CT_DELETE_ORIGIN_DIFFERS] = {
+ .conflict_type = CT_DELETE_ORIGIN_DIFFERS,
+ .conflict_name = "delete_origin_differs",
+ .default_resolver = CR_APPLY_REMOTE,
+ .allowed_resolvers = {[CR_APPLY_REMOTE]=true, [CR_KEEP_LOCAL]=true, [CR_ERROR]=true},
+ },
};
+StaticAssertDecl(lengthof(ConflictInfoMap) == CONFLICT_NUM_TYPES,
+ "array length mismatch");
+
static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
@@ -122,7 +203,7 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictTypeNames[type]),
+ ConflictInfoMap[type].conflict_name),
errdetail_apply_conflict(estate, relinfo, type, searchslot,
localslot, remoteslot, indexoid,
localxmin, localorigin, localts));
@@ -489,3 +570,355 @@ build_index_value_desc(EState *estate, Relation localrel, TupleTableSlot *slot,
return index_value;
}
+
+/*
+ * Report a warning about incomplete conflict detection and resolution if
+ * track_commit_timestamp is disabled.
+ */
+static void
+conf_detection_check_prerequisites(void)
+{
+ if (!track_commit_timestamp)
+ ereport(WARNING,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("conflict detection and resolution could be incomplete due to disabled track_commit_timestamp"),
+ errdetail("Conflict types 'update_origin_differs' and 'delete_origin_differs'"
+ "cannot be detected unless 'track_commit_timestamp' is enabled"));
+}
+
+/*
+ * Validate the conflict type and return the corresponding ConflictType enum.
+ */
+ConflictType
+ValidateConflictType(const char *conflict_type)
+{
+ ConflictType type;
+ bool valid = false;
+
+ /* Check conflict type validity */
+ for (type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ if (pg_strcasecmp(ConflictInfoMap[type].conflict_name, conflict_type) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict type", conflict_type));
+
+ return type;
+}
+
+/*
+ * Validate the conflict type and resolver. It returns an enum ConflictType
+ * corresponding to the conflict type string passed by the caller.
+ */
+ConflictType
+ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver)
+{
+ ConflictType type;
+ ConflictResolver resolver;
+ bool valid = false;
+
+ /* Validate conflict type */
+ type = ValidateConflictType(conflict_type);
+
+ /* Validate the conflict resolver name */
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_resolver) == 0)
+ {
+ valid = true;
+ break;
+ }
+ }
+
+ if (!valid)
+ {
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver", conflict_resolver));
+ }
+
+ valid = ConflictInfoMap[type].allowed_resolvers[resolver];
+
+ if (!valid)
+ ereport(ERROR,
+ errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%s is not a valid conflict resolver for conflict type %s",
+ conflict_resolver,
+ conflict_type));
+
+ return type;
+}
+
+/*
+ * Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
+ * SUBSCRIPTION commands.
+ *
+ * It reports an error if duplicate options are specified.
+ *
+ * Returns a list of conflict types along with their corresponding conflict
+ * resolvers. If 'add_defaults' is true, it appends default resolvers for any
+ * conflict types that have not been explicitly defined by the user.
+ */
+List *
+ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers, bool add_defaults)
+{
+ List *res = NIL;
+ bool already_seen[CONFLICT_NUM_TYPES] = {0};
+ ConflictType type;
+
+ /* Loop through the user provided resolvers */
+ foreach_ptr(DefElem, defel, stmtresolvers)
+ {
+ char *resolver;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ /* Validate the conflict type and resolver */
+ resolver = defGetString(defel);
+ resolver = downcase_truncate_identifier(
+ resolver, strlen(resolver), false);
+ type = ValidateConflictTypeAndResolver(defel->defname,
+ resolver);
+
+ /* Check if the conflict type has already been seen */
+ if (already_seen[type])
+ errorConflictingDefElem(defel, pstate);
+
+ already_seen[type] = true;
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = defel->defname;
+ conftyperesolver->conflict_resolver_name = resolver;
+ res = lappend(res, conftyperesolver);
+ }
+
+ /* Once validation is complete, warn users if prerequisites are not met */
+ if (stmtresolvers)
+ conf_detection_check_prerequisites();
+
+ /*
+ * If add_defaults is true, fill remaining conflict types with default
+ * resolvers.
+ */
+ if (add_defaults)
+ {
+ for (int i = 0; i < CONFLICT_NUM_TYPES; i++)
+ {
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ if (!already_seen[i])
+ {
+ ConflictResolver def_resolver = ConflictInfoMap[i].default_resolver;
+
+ conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+ conftyperesolver->conflict_type_name = ConflictInfoMap[i].conflict_name;
+ conftyperesolver->conflict_resolver_name = ConflictResolverNames[def_resolver];;
+ res = lappend(res, conftyperesolver);
+ }
+ }
+ }
+
+ return res;
+}
+
+/*
+ * Get the list of conflict types and their corresponding default resolvers.
+ */
+List *
+GetDefaultConflictResolvers()
+{
+ List *res = NIL;
+
+ for (ConflictType type = 0; type < CONFLICT_NUM_TYPES; type++)
+ {
+ ConflictTypeResolver *resolver = NULL;
+ ConflictResolver def_resolver = ConflictInfoMap[type].default_resolver;
+
+ /* Allocate memory for each ConflictTypeResolver */
+ resolver = palloc(sizeof(ConflictTypeResolver));
+
+ resolver->conflict_type_name = ConflictInfoMap[type].conflict_name;
+ resolver->conflict_resolver_name = ConflictResolverNames[def_resolver];
+
+ /* Append to the response list */
+ res = lappend(res, resolver);
+ }
+
+ return res;
+}
+
+/*
+ * Update the Subscription's conflict resolvers in pg_subscription_conflict
+ * system catalog for the given conflict types.
+ */
+void
+UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid)
+{
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ bool replaces[Natts_pg_subscription_conflict];
+ HeapTuple oldtup;
+ HeapTuple newtup = NULL;
+ Relation pg_subscription_conflict;
+ char *cur_conflict_res;
+ Datum datum;
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+ memset(replaces, false, sizeof(replaces));
+ memset(values, 0, sizeof(values));
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ /* Set up subid and conflict_type to search in cache */
+ values[Anum_pg_subscription_conflict_confsubid - 1] = ObjectIdGetDatum(subid);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+
+ oldtup = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ values[Anum_pg_subscription_conflict_confsubid - 1],
+ values[Anum_pg_subscription_conflict_conftype - 1]);
+
+ if (!HeapTupleIsValid(oldtup))
+ elog(ERROR, "cache lookup failed for table conflict %s for subid %u",
+ conftyperesolver->conflict_type_name, subid);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ oldtup, Anum_pg_subscription_conflict_confres);
+ cur_conflict_res = TextDatumGetCString(datum);
+
+ /*
+ * Update system catalog only if the new resolver is not same as the
+ * existing one.
+ */
+ if (pg_strcasecmp(cur_conflict_res,
+ conftyperesolver->conflict_resolver_name) != 0)
+ {
+ /* Update the new resolver */
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+ replaces[Anum_pg_subscription_conflict_confres - 1] = true;
+
+ newtup = heap_modify_tuple(oldtup,
+ RelationGetDescr(pg_subscription_conflict),
+ values, nulls, replaces);
+ CatalogTupleUpdate(pg_subscription_conflict,
+ &oldtup->t_self, newtup);
+ heap_freetuple(newtup);
+ }
+
+ ReleaseSysCache(oldtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Reset the conflict resolver for this conflict type to its default setting.
+ */
+void
+ResetSubConflictResolver(Oid subid, char *conflict_type)
+{
+ ConflictType idx;
+ ConflictTypeResolver *conflictResolver = NULL;
+ List *conflictresolver_list = NIL;
+
+ /* Validate the conflict type and get the index */
+ idx = ValidateConflictType(conflict_type);
+ conflictResolver = palloc(sizeof(ConflictTypeResolver));
+ conflictResolver->conflict_type_name = conflict_type;
+
+ /* Get the default resolver for this conflict_type */
+ conflictResolver->conflict_resolver_name =
+ ConflictResolverNames[ConflictInfoMap[idx].default_resolver];
+
+ /* Create a list of conflict resolvers and update in catalog */
+ conflictresolver_list = lappend(conflictresolver_list, conflictResolver);
+ UpdateSubConflictResolvers(conflictresolver_list, subid);
+
+}
+
+/*
+ * Set Conflict Resolvers on the subscription
+ */
+void
+SetSubConflictResolvers(Oid subId, List *conflict_resolvers)
+{
+ Relation pg_subscription_conflict;
+ Datum values[Natts_pg_subscription_conflict];
+ bool nulls[Natts_pg_subscription_conflict];
+ HeapTuple newtup = NULL;
+ Oid conflict_oid;
+
+ pg_subscription_conflict = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /* Prepare to update a tuple */
+ memset(nulls, false, sizeof(nulls));
+
+ /* Iterate over the list of resolvers */
+ foreach_ptr(ConflictTypeResolver, conftyperesolver, conflict_resolvers)
+ {
+ values[Anum_pg_subscription_conflict_confsubid - 1] =
+ ObjectIdGetDatum(subId);
+ values[Anum_pg_subscription_conflict_conftype - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_type_name);
+ values[Anum_pg_subscription_conflict_confres - 1] =
+ CStringGetTextDatum(conftyperesolver->conflict_resolver_name);
+
+ /* Get a new oid and update the tuple into catalog */
+ conflict_oid = GetNewOidWithIndex(pg_subscription_conflict,
+ SubscriptionConflictOidIndexId,
+ Anum_pg_subscription_conflict_oid);
+ values[Anum_pg_subscription_conflict_oid - 1] =
+ ObjectIdGetDatum(conflict_oid);
+ newtup = heap_form_tuple(RelationGetDescr(pg_subscription_conflict),
+ values, nulls);
+ CatalogTupleInsert(pg_subscription_conflict, newtup);
+ heap_freetuple(newtup);
+ }
+
+ table_close(pg_subscription_conflict, RowExclusiveLock);
+}
+
+/*
+ * Remove the subscription conflict resolvers for the subscription id
+ */
+void
+RemoveSubConflictResolvers(Oid subid)
+{
+ Relation rel;
+ HeapTuple tup;
+ TableScanDesc scan;
+ ScanKeyData skey[1];
+
+ rel = table_open(SubscriptionConflictId, RowExclusiveLock);
+
+ /*
+ * Search using the subid to return all conflict resolvers for this
+ * subscription.
+ */
+ ScanKeyInit(&skey[0],
+ Anum_pg_subscription_conflict_confsubid,
+ BTEqualStrategyNumber,
+ F_OIDEQ,
+ ObjectIdGetDatum(subid));
+
+ scan = table_beginscan_catalog(rel, 1, skey);
+
+ /* Iterate through the tuples and delete them */
+ while (HeapTupleIsValid(tup = heap_getnext(scan, ForwardScanDirection)))
+ CatalogTupleDelete(rel, &tup->t_self);
+
+ table_endscan(scan);
+ table_close(rel, RowExclusiveLock);
+}
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 1b47c38..921b1e5 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -395,6 +395,8 @@ static void setupDumpWorker(Archive *AH);
static TableInfo *getRootTableInfo(const TableInfo *tbinfo);
static bool forcePartitionRootLoad(const TableInfo *tbinfo);
static void read_dump_filters(const char *filename, DumpOptions *dopt);
+static bool is_default_resolver(const char *confType, const char *confRes);
+static void destroyConcflictResolverList(SimplePtrList *list);
int
@@ -4830,7 +4832,9 @@ getSubscriptions(Archive *fout)
{
DumpOptions *dopt = fout->dopt;
PQExpBuffer query;
+ PQExpBuffer confQuery;
PGresult *res;
+ PGresult *confRes;
SubscriptionInfo *subinfo;
int i_tableoid;
int i_oid;
@@ -4851,7 +4855,9 @@ getSubscriptions(Archive *fout)
int i_subenabled;
int i_subfailover;
int i,
- ntups;
+ j,
+ ntups,
+ conf_ntuples;
if (dopt->no_subscriptions || fout->remoteVersion < 100000)
return;
@@ -5012,6 +5018,38 @@ getSubscriptions(Archive *fout)
subinfo[i].subfailover =
pg_strdup(PQgetvalue(res, i, i_subfailover));
+ /* Populate conflict type fields using the new query */
+ confQuery = createPQExpBuffer();
+ appendPQExpBuffer(confQuery,
+ "SELECT conftype, confres FROM pg_catalog.pg_subscription_conflict "
+ "WHERE confsubid = %u order by conftype;", subinfo[i].dobj.catId.oid);
+ confRes = ExecuteSqlQuery(fout, confQuery->data, PGRES_TUPLES_OK);
+
+ conf_ntuples = PQntuples(confRes);
+
+ /* Initialize pointers in the list to NULL */
+ subinfo[i].conflict_resolvers = (SimplePtrList)
+ {
+ .head = NULL, .tail = NULL
+
+ };
+
+ /* Store conflict types and resolvers from the query result in subinfo */
+ for (j = 0; j < conf_ntuples; j++)
+ {
+ /* Create the ConflictTypeResolver node */
+ ConflictTypeResolver *conftyperesolver = palloc(sizeof(ConflictTypeResolver));
+
+ conftyperesolver->conflict_type = pg_strdup(PQgetvalue(confRes, j, 0));
+ conftyperesolver->resolver = pg_strdup(PQgetvalue(confRes, j, 1));
+
+ /* Append the node to subinfo's list */
+ simple_ptr_list_append(&subinfo[i].conflict_resolvers, conftyperesolver);
+ }
+
+ PQclear(confRes);
+ destroyPQExpBuffer(confQuery);
+
/* Decide whether we want to dump it */
selectDumpableObject(&(subinfo[i].dobj), fout);
}
@@ -5222,7 +5260,43 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
appendPQExpBufferStr(publications, fmtId(pubnames[i]));
}
- appendPQExpBuffer(query, " PUBLICATION %s WITH (connect = false, slot_name = ", publications->data);
+ appendPQExpBuffer(query, " PUBLICATION %s ", publications->data);
+
+ /* Add conflict resolvers, if any */
+ if (fout->remoteVersion >= 180000)
+ {
+ bool first_resolver = true;
+ SimplePtrListCell *cell = NULL;
+ ConflictTypeResolver *conftyperesolver = NULL;
+
+ for (cell = subinfo->conflict_resolvers.head; cell; cell = cell->next)
+ {
+ conftyperesolver = (ConflictTypeResolver *) cell->ptr;
+
+ if (!is_default_resolver(conftyperesolver->conflict_type,
+ conftyperesolver->resolver))
+ {
+ if (first_resolver)
+ {
+ appendPQExpBuffer(query, "CONFLICT RESOLVER (%s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ first_resolver = false;
+ }
+ else
+ appendPQExpBuffer(query, ", %s = '%s'",
+ conftyperesolver->conflict_type,
+ conftyperesolver->resolver);
+ }
+ }
+
+ /* If there was at least one resolver, close the braces */
+ if (!first_resolver)
+ appendPQExpBufferStr(query, ") ");
+ }
+
+ appendPQExpBuffer(query, "WITH (connect = false, slot_name = ");
+
if (subinfo->subslotname)
appendStringLiteralAH(query, subinfo->subslotname, fout);
else
@@ -5315,6 +5389,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo)
NULL, subinfo->rolname,
subinfo->dobj.catId, 0, subinfo->dobj.dumpId);
+ /* Clean-up the conflict_resolver list */
+ destroyConcflictResolverList((SimplePtrList *) &subinfo->conflict_resolvers);
+
destroyPQExpBuffer(publications);
free(pubnames);
@@ -19051,3 +19128,48 @@ read_dump_filters(const char *filename, DumpOptions *dopt)
filter_free(&fstate);
}
+
+/*
+ * is_default_resolver - checks if the given resolver is the default for the
+ * specified conflict type.
+ */
+static bool
+is_default_resolver(const char *confType, const char *confRes)
+{
+ /*
+ * The default resolvers for each conflict type are taken from the
+ * predefined mapping ConflictTypeDefaultResolvers[] in conflict.c.
+ *
+ * Only modify these defaults if the corresponding values in conflict.c
+ * are changed.
+ */
+
+ if (strcmp(confType, "insert_exists") == 0 ||
+ strcmp(confType, "update_exists") == 0)
+ return strcmp(confRes, "error") == 0;
+ else if (strcmp(confType, "update_missing") == 0 ||
+ strcmp(confType, "delete_missing") == 0)
+ return strcmp(confRes, "skip") == 0;
+ else if (strcmp(confType, "update_origin_differs") == 0 ||
+ strcmp(confType, "delete_origin_differs") == 0)
+ return strcmp(confRes, "apply_remote") == 0;
+
+ return false;
+}
+
+/*
+ * destroyConflictResolverList - frees up the SimplePtrList containing
+ * cells pointing to struct ConflictTypeResolver nodes.
+ */
+static void
+destroyConcflictResolverList(SimplePtrList *conflictlist)
+{
+ SimplePtrListCell *cell = NULL;
+
+ /* Free the list items */
+ for (cell = conflictlist->head; cell; cell = cell->next)
+ pfree((ConflictTypeResolver *) cell->ptr);
+
+ /* Destroy the pointer list */
+ simple_ptr_list_destroy(conflictlist);
+}
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 9f907ed..35a23eb 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -655,6 +655,13 @@ typedef struct _PublicationSchemaInfo
/*
* The SubscriptionInfo struct is used to represent subscription.
*/
+
+typedef struct _ConflictTypeResolver
+{
+ const char *conflict_type;
+ const char *resolver;
+} ConflictTypeResolver;
+
typedef struct _SubscriptionInfo
{
DumpableObject dobj;
@@ -673,6 +680,7 @@ typedef struct _SubscriptionInfo
char *suborigin;
char *suboriginremotelsn;
char *subfailover;
+ SimplePtrList conflict_resolvers;
} SubscriptionInfo;
/*
diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl
index ab6c830..ec8b446 100644
--- a/src/bin/pg_dump/t/002_pg_dump.pl
+++ b/src/bin/pg_dump/t/002_pg_dump.pl
@@ -3012,9 +3012,10 @@ my %tests = (
create_order => 50,
create_sql => 'CREATE SUBSCRIPTION sub3
CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1
+ CONFLICT RESOLVER (insert_exists = "apply_remote", update_missing = "error")
WITH (connect = false, origin = any);',
regexp => qr/^
- \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3');\E
+ \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'error') WITH (connect = false, slot_name = 'sub3');\E
/xm,
like => { %full_runs, section_post_data => 1, },
},
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 6a36c91..636c507 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6545,9 +6545,10 @@ describeSubscriptions(const char *pattern, bool verbose)
PQExpBufferData buf;
PGresult *res;
printQueryOpt myopt = pset.popt;
+ char **footers = NULL;
static const bool translate_columns[] = {false, false, false, false,
false, false, false, false, false, false, false, false, false, false,
- false};
+ false, false, false, false, false, false, false};
if (pset.sversion < 100000)
{
@@ -6627,12 +6628,13 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBuffer(&buf,
", subskiplsn AS \"%s\"\n",
gettext_noop("Skip LSN"));
+
}
/* Only display subscriptions in current database. */
appendPQExpBufferStr(&buf,
- "FROM pg_catalog.pg_subscription\n"
- "WHERE subdbid = (SELECT oid\n"
+ "FROM pg_catalog.pg_subscription s\n"
+ "WHERE s.subdbid = (SELECT oid\n"
" FROM pg_catalog.pg_database\n"
" WHERE datname = pg_catalog.current_database())");
@@ -6648,7 +6650,6 @@ describeSubscriptions(const char *pattern, bool verbose)
appendPQExpBufferStr(&buf, "ORDER BY 1;");
res = PSQLexec(buf.data);
- termPQExpBuffer(&buf);
if (!res)
return false;
@@ -6657,9 +6658,118 @@ describeSubscriptions(const char *pattern, bool verbose)
myopt.translate_columns = translate_columns;
myopt.n_translate_columns = lengthof(translate_columns);
+ if (verbose && pset.sversion >= 180000)
+ {
+ PGresult *result;
+ int i,
+ tuples,
+ max_len = 0;
+ char *prev_subname = NULL;
+
+ printfPQExpBuffer(&buf,
+ "SELECT s.subname, c.conftype, c.confres \n"
+ " FROM pg_catalog.pg_subscription_conflict c \n"
+ " JOIN pg_catalog.pg_subscription s \n"
+ " ON c.confsubid = s.oid \n"
+ " WHERE s.subdbid = (SELECT oid\n"
+ " FROM pg_catalog.pg_database\n"
+ " WHERE datname = pg_catalog.current_database())");
+
+ if (!validateSQLNamePattern(&buf, pattern, true, false,
+ NULL, "s.subname", NULL,
+ NULL,
+ NULL, 1))
+ {
+ termPQExpBuffer(&buf);
+ return false;
+ }
+
+ appendPQExpBufferStr(&buf, "ORDER BY s.subname, c.conftype");
+
+ result = PSQLexec(buf.data);
+ if (!result)
+ {
+ termPQExpBuffer(&buf);
+ return false;
+ }
+ else
+ tuples = PQntuples(result);
+
+ /* Calculate the maximum length of the conflict type (conftype) */
+ for (i = 0; i < tuples; i++)
+ {
+ int len = strlen(PQgetvalue(result, i, 1));
+ if (len > max_len)
+ max_len = len;
+ }
+
+ if (tuples > 0)
+ {
+ /*
+ * Allocate memory for footers. Size of footers will be
+ * 1 (for storing "Conflict Resolvers" string) + conflict
+ * resolver count + subscription count + 1 (for storing NULL).
+ * Subscription count is determined by dividing the number of
+ * conflict resolvers by 6. This needs to be changed when
+ * CONFLICT_NUM_RESOLVERS in conflict.h changes
+ */
+ int subscription_count = tuples / 6;
+ int footer_idx = 1;
+
+ footers = (char **) pg_malloc((1 + tuples + subscription_count + 1)
+ * sizeof(char *));
+ footers[0] = pg_strdup(_("\nConflict Resolvers:"));
+
+ for (i = 0; i < tuples; i++)
+ {
+ const char *subname = PQgetvalue(result, i, 0);
+ const char *conftype = PQgetvalue(result, i, 1);
+ const char *confres = PQgetvalue(result, i, 2);
+
+ /*
+ * Print the subscription name only if it changes
+ * (group by subname).
+ */
+ if (!prev_subname || strcmp(prev_subname, subname) != 0)
+ {
+ printfPQExpBuffer(&buf, "\n(%s)", subname);
+ footers[footer_idx++] = pg_strdup(buf.data);
+ prev_subname = pg_strdup(subname);
+ }
+
+ /* Adjust the padding to align the "=" */
+ printfPQExpBuffer(&buf, "%s%*s = \"%s\"",
+ conftype,
+ (int)(max_len - strlen(conftype) + 2),
+ "",
+ confres);
+
+ footers[footer_idx++] = pg_strdup(buf.data);
+ }
+
+ footers[footer_idx] = NULL;
+ myopt.footers = footers;
+ }
+
+ PQclear(result);
+ }
+
printQuery(res, &myopt, pset.queryFout, false, pset.logfile);
+ termPQExpBuffer(&buf);
PQclear(res);
+
+ /* Free the memory allocated for the footer */
+ if (footers)
+ {
+ char **footer = NULL;
+
+ for (footer = footers; *footer; footer++)
+ pg_free(*footer);
+
+ pg_free(footers);
+ }
+
return true;
}
diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c
index 1be0056..a083e1e 100644
--- a/src/bin/psql/tab-complete.in.c
+++ b/src/bin/psql/tab-complete.in.c
@@ -2266,7 +2266,7 @@ match_previous_words(int pattern_id,
else if (Matches("ALTER", "SUBSCRIPTION", MatchAny))
COMPLETE_WITH("CONNECTION", "ENABLE", "DISABLE", "OWNER TO",
"RENAME TO", "REFRESH PUBLICATION", "SET", "SKIP (",
- "ADD PUBLICATION", "DROP PUBLICATION");
+ "ADD PUBLICATION", "DROP PUBLICATION", "CONFLICT RESOLVER (");
/* ALTER SUBSCRIPTION <name> REFRESH PUBLICATION */
else if (Matches("ALTER", "SUBSCRIPTION", MatchAny, MatchAnyN, "REFRESH", "PUBLICATION"))
COMPLETE_WITH("WITH (");
@@ -2297,6 +2297,19 @@ match_previous_words(int pattern_id,
else if (Matches("ALTER", "SUBSCRIPTION", MatchAny, MatchAnyN,
"ADD|DROP|SET", "PUBLICATION", MatchAny, "WITH", "("))
COMPLETE_WITH("copy_data", "refresh");
+ /* ALTER SUBSCRIPTION <name> CONFLICT RESOLVER (<conflict_type> = <conflict_resolver> [, ..]) */
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESOLVER", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
+ /* ALTER SUBSCRIPTION <name> RESET CONFLICT RESOLVER ALL|FOR <conflict_type> */
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESET", "CONFLICT", "RESOLVER"))
+ COMPLETE_WITH("ALL", "FOR (");
+ else if (HeadMatches("ALTER", "SUBSCRIPTION", MatchAny) &&
+ TailMatches("RESOLVER", "FOR", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
/* ALTER SCHEMA <name> */
else if (Matches("ALTER", "SCHEMA", MatchAny))
@@ -3668,8 +3681,13 @@ match_previous_words(int pattern_id,
{
/* complete with nothing here as this refers to remote publications */
}
+ /* Complete "CREATE SUBSCRIPTION <name> ... CONFLICT RESOLVER ( <confoptions> ) */
else if (Matches("CREATE", "SUBSCRIPTION", MatchAnyN, "PUBLICATION", MatchAny))
- COMPLETE_WITH("WITH (");
+ COMPLETE_WITH("CONFLICT RESOLVER", "WITH (");
+ else if (HeadMatches("CREATE", "SUBSCRIPTION") && TailMatches("CONFLICT", "RESOLVER", "("))
+ COMPLETE_WITH("insert_exists", "update_origin_differs", "update_exists",
+ "update_missing", "delete_origin_differs", "delete_missing");
+
/* Complete "CREATE SUBSCRIPTION <name> ... WITH ( <opt>" */
else if (Matches("CREATE", "SUBSCRIPTION", MatchAnyN, "WITH", "("))
COMPLETE_WITH("binary", "connect", "copy_data", "create_slot",
diff --git a/src/include/catalog/Makefile b/src/include/catalog/Makefile
index 167f91a..f2611c1 100644
--- a/src/include/catalog/Makefile
+++ b/src/include/catalog/Makefile
@@ -81,7 +81,8 @@ CATALOG_HEADERS := \
pg_publication_namespace.h \
pg_publication_rel.h \
pg_subscription.h \
- pg_subscription_rel.h
+ pg_subscription_rel.h \
+ pg_subscription_conflict.h
GENERATED_HEADERS := $(CATALOG_HEADERS:%.h=%_d.h)
diff --git a/src/include/catalog/meson.build b/src/include/catalog/meson.build
index f70d1da..959e1d9 100644
--- a/src/include/catalog/meson.build
+++ b/src/include/catalog/meson.build
@@ -69,6 +69,7 @@ catalog_headers = [
'pg_publication_rel.h',
'pg_subscription.h',
'pg_subscription_rel.h',
+ 'pg_subscription_conflict.h',
]
# The .dat files we need can just be listed alphabetically.
diff --git a/src/include/catalog/pg_subscription_conflict.h b/src/include/catalog/pg_subscription_conflict.h
new file mode 100644
index 0000000..15f02a1
--- /dev/null
+++ b/src/include/catalog/pg_subscription_conflict.h
@@ -0,0 +1,55 @@
+/* -------------------------------------------------------------------------
+ *
+ * pg_subscription_conflict.h
+ * definition of the "subscription conflict resolver" system
+ * catalog (pg_subscription_conflict)
+ *
+ * Portions Copyright (c) 1996-2024, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_subscription_conflict.h
+ *
+ * NOTES
+ * The Catalog.pm module reads this file and derives schema
+ * information.
+ *
+ * -------------------------------------------------------------------------
+ */
+#ifndef PG_SUBSCRIPTION_CONFLICT_H
+#define PG_SUBSCRIPTION_CONFLICT_H
+
+#include "catalog/genbki.h"
+#include "catalog/pg_subscription_conflict_d.h"
+
+/* ----------------
+ * pg_subscription_conflict definition. cpp turns this into
+ * typedef struct FormData_pg_subscription_conflict
+ * ----------------
+ */
+CATALOG(pg_subscription_conflict,8881,SubscriptionConflictId)
+{
+ Oid oid; /* OID of the object itself */
+ Oid confsubid BKI_LOOKUP(pg_subscription); /* OID of subscription */
+
+#ifdef CATALOG_VARLEN /* variable-length fields start here */
+ text conftype BKI_FORCE_NOT_NULL; /* conflict type */
+ text confres BKI_FORCE_NOT_NULL; /* conflict resolver */
+#endif
+} FormData_pg_subscription_conflict;
+
+/* ----------------
+ * Form_pg_subscription_conflict corresponds to a pointer to a row with
+ * the format of pg_subscription_conflict relation.
+ * ----------------
+ */
+typedef FormData_pg_subscription_conflict *Form_pg_subscription_conflict;
+
+DECLARE_TOAST(pg_subscription_conflict, 8882, 8883);
+
+DECLARE_UNIQUE_INDEX_PKEY(pg_subscription_conflict_oid_index, 8884, SubscriptionConflictOidIndexId, pg_subscription_conflict, btree(oid oid_ops));
+
+DECLARE_UNIQUE_INDEX(pg_subscription_conflict_confsubid_conftype_index, 8885, SubscriptionConflictSubIdTypeIndexId, pg_subscription_conflict, btree(confsubid oid_ops, conftype text_ops));
+
+MAKE_SYSCACHE(SUBSCRIPTIONCONFLMAP, pg_subscription_conflict_confsubid_conftype_index, 256);
+
+#endif /* PG_SUBSCRIPTION_CONFLICT_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 5b62df3..162b205 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -4214,6 +4214,7 @@ typedef struct CreateSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict resolvers */
} CreateSubscriptionStmt;
typedef enum AlterSubscriptionType
@@ -4226,6 +4227,9 @@ typedef enum AlterSubscriptionType
ALTER_SUBSCRIPTION_REFRESH,
ALTER_SUBSCRIPTION_ENABLED,
ALTER_SUBSCRIPTION_SKIP,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVERS,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET,
+ ALTER_SUBSCRIPTION_CONFLICT_RESOLVER_RESET_ALL,
} AlterSubscriptionType;
typedef struct AlterSubscriptionStmt
@@ -4236,6 +4240,9 @@ typedef struct AlterSubscriptionStmt
char *conninfo; /* Connection string to publisher */
List *publication; /* One or more publication to subscribe to */
List *options; /* List of DefElem nodes */
+ List *resolvers; /* List of conflict type names and resolver
+ * names */
+ char *conflict_type_name; /* The conflict type name to be reset */
} AlterSubscriptionStmt;
typedef struct DropSubscriptionStmt
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 899d64a..e62b0c6 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -376,6 +376,7 @@ PG_KEYWORD("repeatable", REPEATABLE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replace", REPLACE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("replica", REPLICA, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("reset", RESET, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("resolver", RESOLVER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restart", RESTART, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("restrict", RESTRICT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("return", RETURN, UNRESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c759677..c5865a1 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -1,6 +1,6 @@
/*-------------------------------------------------------------------------
* conflict.h
- * Exports for conflicts logging.
+ * Exports for conflicts logging and resolvers configuration.
*
* Copyright (c) 2024, PostgreSQL Global Development Group
*
@@ -10,6 +10,7 @@
#define CONFLICT_H
#include "nodes/execnodes.h"
+#include "parser/parse_node.h"
#include "utils/timestamp.h"
/*
@@ -50,6 +51,41 @@ typedef enum
#define CONFLICT_NUM_TYPES (CT_DELETE_MISSING + 1)
+/*
+ * Conflict resolvers that can be used to resolve various conflicts.
+ *
+ * See ConflictTypeResolverMap in conflict.c to find out which all
+ * resolvers are supported for each conflict type.
+ */
+typedef enum ConflictResolver
+{
+ /* Apply the remote change */
+ CR_APPLY_REMOTE,
+
+ /* Keep the local change */
+ CR_KEEP_LOCAL,
+
+ /* Apply the remote change; skip if it cannot be applied */
+ CR_APPLY_OR_SKIP,
+
+ /* Apply the remote change; emit error if it cannot be applied */
+ CR_APPLY_OR_ERROR,
+
+ /* Skip applying the change */
+ CR_SKIP,
+
+ /* Error out */
+ CR_ERROR,
+} ConflictResolver;
+
+#define CONFLICT_NUM_RESOLVERS (CR_ERROR + 1)
+
+typedef struct ConflictTypeResolver
+{
+ const char *conflict_type_name;
+ const char *conflict_resolver_name;
+} ConflictTypeResolver;
+
extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
TransactionId *xmin,
RepOriginId *localorigin,
@@ -62,5 +98,16 @@ extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin, TimestampTz localts);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
+extern void SetSubConflictResolvers(Oid subId, List *resolvers);
+extern void RemoveSubConflictResolvers(Oid confid);
+extern List *ParseAndGetSubConflictResolvers(ParseState *pstate,
+ List *stmtresolvers,
+ bool add_defaults);
+extern void UpdateSubConflictResolvers(List *conflict_resolvers, Oid subid);
+extern ConflictType ValidateConflictType(const char *conflict_type);
+extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
+ const char *conflict_resolver);
+extern List *GetDefaultConflictResolvers(void);
+extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
#endif
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb89..42bf2cc 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -266,3 +266,4 @@ NOTICE: checking pg_subscription {subdbid} => pg_database {oid}
NOTICE: checking pg_subscription {subowner} => pg_authid {oid}
NOTICE: checking pg_subscription_rel {srsubid} => pg_subscription {oid}
NOTICE: checking pg_subscription_rel {srrelid} => pg_class {oid}
+NOTICE: checking pg_subscription_conflict {confsubid} => pg_subscription {oid}
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index 17d48b1..bcf4bcd 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -120,7 +120,16 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | none | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub4)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
\dRs+ regress_testsub4
@@ -128,7 +137,16 @@ ALTER SUBSCRIPTION regress_testsub4 SET (origin = any);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub4 | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub4)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
DROP SUBSCRIPTION regress_testsub3;
DROP SUBSCRIPTION regress_testsub4;
@@ -149,7 +167,16 @@ ERROR: invalid connection string syntax: missing "=" after "foobar" in connecti
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
ALTER SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist2';
@@ -161,7 +188,16 @@ ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | f | t | f | off | dbname=regress_doesnotexist2 | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (password_required = true);
ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = false);
@@ -180,7 +216,16 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345');
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/12345
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
-- ok - with lsn = NONE
ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE);
@@ -192,7 +237,16 @@ ERROR: invalid WAL location (LSN): 0/0
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist2 | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
@@ -227,7 +281,16 @@ HINT: Available values: local, remote_write, remote_apply, on, off.
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+------------------------------+----------
regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | off | d | f | any | t | f | f | local | dbname=regress_doesnotexist2 | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub_foo)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
-- rename back to keep the rest simple
ALTER SUBSCRIPTION regress_testsub_foo RENAME TO regress_testsub;
@@ -259,7 +322,16 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | t | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (binary = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -268,7 +340,16 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
DROP SUBSCRIPTION regress_testsub;
-- fail - streaming must be boolean or 'parallel'
@@ -283,7 +364,16 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
\dRs+
@@ -291,7 +381,16 @@ ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (streaming = false);
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
@@ -300,7 +399,16 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
-- fail - publication already exists
ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub WITH (refresh = false);
@@ -318,7 +426,16 @@ ERROR: publication "testpub1" is already in subscription "regress_testsub"
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
-- fail - publication used more than once
ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub1 WITH (refresh = false);
@@ -336,7 +453,16 @@ ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (ref
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
DROP SUBSCRIPTION regress_testsub;
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION mypub
@@ -375,7 +501,16 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
-- we can alter streaming when two_phase enabled
ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
@@ -384,7 +519,16 @@ ALTER SUBSCRIPTION regress_testsub SET (streaming = true);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
@@ -397,7 +541,169 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+ERROR: foo is not a valid conflict type
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+ERROR: foo is not a valid conflict resolver
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+ERROR: apply_remote is not a valid conflict resolver for conflict type update_missing
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+ERROR: conflicting or redundant options
+LINE 1: ... CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exi...
+ ^
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflict types 'update_origin_differs' and 'delete_origin_differs'cannot be detected unless 'track_commit_timestamp' is enabled
+WARNING: subscription was created, but is not connected
+HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and refresh the subscription.
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "keep_local"
+insert_exists = "keep_local"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+ERROR: foo is not a valid conflict type
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+ERROR: foo is not a valid conflict resolver
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+ERROR: conflicting or redundant options
+LINE 1: ...ONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exi...
+ ^
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflict types 'update_origin_differs' and 'delete_origin_differs'cannot be detected unless 'track_commit_timestamp' is enabled
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "keep_local"
+insert_exists = "apply_remote"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+WARNING: conflict detection and resolution could be incomplete due to disabled track_commit_timestamp
+DETAIL: Conflict types 'update_origin_differs' and 'delete_origin_differs'cannot be detected unless 'track_commit_timestamp' is enabled
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "error"
+delete_origin_differs = "keep_local"
+insert_exists = "apply_remote"
+update_exists = "keep_local"
+update_missing = "skip"
+update_origin_differs = "error"
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+ERROR: foo is not a valid conflict type
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "error"
+delete_origin_differs = "keep_local"
+insert_exists = "error"
+update_exists = "keep_local"
+update_missing = "skip"
+update_origin_differs = "error"
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
+-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
@@ -413,7 +719,16 @@ HINT: To initiate replication, you must manually create the replication slot, e
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
\dRs+
@@ -421,7 +736,16 @@ ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true);
Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Synchronous commit | Conninfo | Skip LSN
-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------------------+-----------------------------+----------
regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | t | any | t | f | f | off | dbname=regress_doesnotexist | 0/0
-(1 row)
+
+Conflict Resolvers:
+
+(regress_testsub)
+delete_missing = "skip"
+delete_origin_differs = "apply_remote"
+insert_exists = "error"
+update_exists = "error"
+update_missing = "skip"
+update_origin_differs = "apply_remote"
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 007c9e7..3f08869 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -272,6 +272,69 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB
ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
DROP SUBSCRIPTION regress_testsub;
+-- fail - invalid conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (foo = 'keep_local') WITH (connect = false);
+
+-- fail - invalid conflict resolver
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = foo) WITH (connect = false);
+
+-- fail - invalid resolver for that conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub
+ CONFLICT RESOLVER (update_missing = 'apply_remote') WITH (connect = false);
+
+-- fail - duplicate conflict type
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', insert_exists = 'keep_local');
+
+-- creating subscription with no explicit conflict resolvers should
+-- configure default conflict resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
+-- ok - valid conflict types and resolvers
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub CONFLICT RESOLVER (insert_exists = 'keep_local', update_missing = 'skip', delete_origin_differs = 'keep_local' ) WITH (connect = false);
+
+--check if above are configured; for non specified conflict types, default resolvers should be seen
+\dRs+
+
+-- fail - altering with invalid conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (foo = 'keep_local');
+
+-- fail - altering with invalid conflict resolver
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'foo');
+
+-- fail - altering with duplicate conflict type
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', insert_exists = 'apply_remote');
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (insert_exists = 'apply_remote', update_missing = 'skip', delete_origin_differs = 'keep_local' );
+
+\dRs+
+
+-- ok - valid conflict types and resolvers
+ALTER SUBSCRIPTION regress_testsub CONFLICT RESOLVER (update_exists = 'keep_local', delete_missing = 'error', update_origin_differs = 'error');
+
+\dRs+
+
+-- fail - reset with an invalid conflit type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'foo';
+
+-- ok - valid conflict type
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER for 'insert_exists';
+
+\dRs+
+
+-- ok - reset ALL
+ALTER SUBSCRIPTION regress_testsub RESET CONFLICT RESOLVER ALL;
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
-- fail - disable_on_error must be boolean
CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, disable_on_error = foo);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index a65e1c0..bf1ea95 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -468,7 +468,9 @@ ConditionVariableMinimallyPadded
ConditionalStack
ConfigData
ConfigVariable
+ConflictResolver
ConflictType
+ConflictTypeResolver
ConnCacheEntry
ConnCacheKey
ConnParams
@@ -863,6 +865,7 @@ FormData_pg_statistic
FormData_pg_statistic_ext
FormData_pg_statistic_ext_data
FormData_pg_subscription
+FormData_pg_subscription_conflict
FormData_pg_subscription_rel
FormData_pg_tablespace
FormData_pg_transform
--
1.8.3.1
v17-0002-Conflict-resolvers-for-insert-update-and-delete.patchapplication/octet-stream; name=v17-0002-Conflict-resolvers-for-insert-update-and-delete.patchDownload
From 1618a40ea46084112b2f7e29b0bfe2ba478721e6 Mon Sep 17 00:00:00 2001
From: Ajin Cherian <itsajin@gmail.com>
Date: Fri, 18 Oct 2024 00:38:55 -0400
Subject: [PATCH v17 2/4] Conflict resolvers for insert, update, and delete
This patch introduces support for handling conflicts with the following resolutions:
- For INSERT conflicts:
- insert_exists: apply_remote, keep_local, error
- For UPDATE conflicts:
- update_origin_differs: apply_remote, keep_local, error
- update_missing: apply_or_skip, apply_or_error, skip, error
- For DELETE conflicts:
- delete_missing: skip, error
- delete_origin_differs: apply_remote, keep_local, error
---
src/backend/executor/execReplication.c | 97 ++-
src/backend/replication/logical/conflict.c | 230 ++++--
src/backend/replication/logical/worker.c | 374 +++++++---
src/include/executor/executor.h | 5 +-
src/include/replication/conflict.h | 12 +-
src/test/subscription/meson.build | 1 +
src/test/subscription/t/034_conflict_resolver.pl | 877 +++++++++++++++++++++++
7 files changed, 1442 insertions(+), 154 deletions(-)
create mode 100644 src/test/subscription/t/034_conflict_resolver.pl
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c
index 54025c9..27d6e98 100644
--- a/src/backend/executor/execReplication.c
+++ b/src/backend/executor/execReplication.c
@@ -550,14 +550,76 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
TransactionId xmin;
GetTupleTransactionInfo(conflictslot, &xmin, &origin, &committs);
- ReportApplyConflict(estate, resultRelInfo, ERROR, type,
+ ReportApplyConflict(estate, resultRelInfo, type, CR_ERROR,
searchslot, conflictslot, remoteslot,
- uniqueidx, xmin, origin, committs);
+ uniqueidx, xmin, origin, committs, false);
}
}
}
/*
+ * Check the unique indexes for conflicts. Return true on finding the
+ * first conflict itself.
+ * If the configured resolver is in favour of apply, give the conflicted
+ * tuple information in conflictslot.
+ */
+static bool
+has_conflicting_tuple(Oid subid, ConflictType type,
+ ResultRelInfo *resultRelInfo, EState *estate,
+ TupleTableSlot *slot, TupleTableSlot **conflictslot)
+{
+ ConflictResolver resolver;
+ bool apply_remote = false;
+ Relation rel = resultRelInfo->ri_RelationDesc;
+ List *conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
+
+ /* ASSERT if called for any conflict type other than insert_exists */
+ Assert(type == CT_INSERT_EXISTS);
+
+ /*
+ * Get the configured resolver and determine if remote changes should be
+ * applied.
+ */
+ resolver = GetConflictResolver(subid, type, rel, NULL, &apply_remote);
+
+ /*
+ * Proceed to find conflict if the resolver is set to a non-default value;
+ * if the resolver is 'ERROR' (default), the caller will handle it.
+ */
+ if (resolver == CR_ERROR)
+ return false;
+
+ /* Check all the unique indexes for a conflict */
+ foreach_oid(uniqueidx, conflictindexes)
+ {
+ /* Return to caller for resolutions if any conflict is found */
+ if (FindConflictTuple(resultRelInfo, estate, uniqueidx, slot,
+ &(*conflictslot)))
+ {
+ RepOriginId origin;
+ TimestampTz committs;
+ TransactionId xmin;
+
+ GetTupleTransactionInfo(*conflictslot, &xmin, &origin, &committs);
+ ReportApplyConflict(estate, resultRelInfo, type, resolver,
+ NULL, *conflictslot, slot, uniqueidx,
+ xmin, origin, committs, apply_remote);
+
+ /* Nothing to apply, free the resources */
+ if (!apply_remote)
+ {
+ ExecDropSingleTupleTableSlot(*conflictslot);
+ *conflictslot = NULL;
+ }
+
+ return true;
+ }
+ }
+
+ return false;
+}
+
+/*
* Insert tuple represented in the slot to the relation, update the indexes,
* and execute any constraints and per-row triggers.
*
@@ -565,7 +627,8 @@ CheckAndReportConflict(ResultRelInfo *resultRelInfo, EState *estate,
*/
void
ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot)
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictslot, Oid subid)
{
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
@@ -601,6 +664,19 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
+ /*
+ * Check for conflict and return to caller for resolution, if found.
+ *
+ * XXX In case there are no conflicts, a non-default 'insert_exists'
+ * resolver adds overhead by performing an extra scan here. However,
+ * this approach avoids the extra work needed to rollback/delete the
+ * inserted tuple if a conflict is detected after insertion with a
+ * non-default resolution set.
+ */
+ if (has_conflicting_tuple(subid, CT_INSERT_EXISTS, resultRelInfo,
+ estate, slot, &(*conflictslot)))
+ return;
+
/* OK, store the tuple and create index entries for it */
simple_table_tuple_insert(resultRelInfo->ri_RelationDesc, slot);
@@ -615,13 +691,14 @@ ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
/*
* Checks the conflict indexes to fetch the conflicting local tuple
- * and reports the conflict. We perform this check here, instead of
- * performing an additional index scan before the actual insertion and
- * reporting the conflict if any conflicting tuples are found. This is
- * to avoid the overhead of executing the extra scan for each INSERT
- * operation, even when no conflict arises, which could introduce
- * significant overhead to replication, particularly in cases where
- * conflicts are rare.
+ * and reports the conflict. We perform this check here again to -
+ *
+ * a) optimize the default case where the resolution for
+ * 'insert_exists' is set to 'ERROR' by skipping the scan when there
+ * is no conflict.
+ *
+ * b) catch and report any conflict that might have been missed during
+ * the pre-insertion scan in has_conflicting_tuple().
*
* XXX OTOH, this could lead to clean-up effort for dead tuples added
* in heap and index in case of conflicts. But as conflicts shouldn't
diff --git a/src/backend/replication/logical/conflict.c b/src/backend/replication/logical/conflict.c
index cb6d50c..6bf9f0a 100644
--- a/src/backend/replication/logical/conflict.c
+++ b/src/backend/replication/logical/conflict.c
@@ -24,9 +24,9 @@
#include "parser/scansup.h"
#include "pgstat.h"
#include "replication/conflict.h"
+#include "replication/worker_internal.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
-#include "replication/worker_internal.h"
#include "storage/lmgr.h"
#include "utils/lsyscache.h"
#include "utils/syscache.h"
@@ -117,12 +117,14 @@ static int errcode_apply_conflict(ConflictType type);
static int errdetail_apply_conflict(EState *estate,
ResultRelInfo *relinfo,
ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
RepOriginId localorigin,
- TimestampTz localts);
+ TimestampTz localts,
+ bool apply_remote);
static char *build_tuple_value_details(EState *estate, ResultRelInfo *relinfo,
ConflictType type,
TupleTableSlot *searchslot,
@@ -165,8 +167,8 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
}
/*
- * This function is used to report a conflict while applying replication
- * changes.
+ * This function is used to report a conflict and resolution applied while
+ * applying replication changes.
*
* 'searchslot' should contain the tuple used to search the local tuple to be
* updated or deleted.
@@ -185,13 +187,22 @@ GetTupleTransactionInfo(TupleTableSlot *localslot, TransactionId *xmin,
* that we can fetch and display the conflicting key value.
*/
void
-ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
+
{
Relation localrel = relinfo->ri_RelationDesc;
+ int elevel;
+
+ if (resolver == CR_ERROR ||
+ (resolver == CR_APPLY_OR_ERROR && !apply_remote))
+ elevel = ERROR;
+ else
+ elevel = LOG;
Assert(!OidIsValid(indexoid) ||
CheckRelationOidLockedByMe(indexoid, RowExclusiveLock, true));
@@ -200,13 +211,14 @@ ReportApplyConflict(EState *estate, ResultRelInfo *relinfo, int elevel,
ereport(elevel,
errcode_apply_conflict(type),
- errmsg("conflict detected on relation \"%s.%s\": conflict=%s",
+ errmsg("conflict detected on relation \"%s.%s\": conflict=%s, resolution=%s.",
get_namespace_name(RelationGetNamespace(localrel)),
RelationGetRelationName(localrel),
- ConflictInfoMap[type].conflict_name),
- errdetail_apply_conflict(estate, relinfo, type, searchslot,
+ ConflictInfoMap[type].conflict_name,
+ ConflictResolverNames[resolver]),
+ errdetail_apply_conflict(estate, relinfo, type, resolver, searchslot,
localslot, remoteslot, indexoid,
- localxmin, localorigin, localts));
+ localxmin, localorigin, localts, apply_remote));
}
/*
@@ -274,17 +286,25 @@ errcode_apply_conflict(ConflictType type)
*/
static int
errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
- ConflictType type, TupleTableSlot *searchslot,
- TupleTableSlot *localslot, TupleTableSlot *remoteslot,
- Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts)
+ ConflictType type, ConflictResolver resolver,
+ TupleTableSlot *searchslot, TupleTableSlot *localslot,
+ TupleTableSlot *remoteslot, Oid indexoid,
+ TransactionId localxmin, RepOriginId localorigin,
+ TimestampTz localts, bool apply_remote)
{
StringInfoData err_detail;
char *val_desc;
char *origin_name;
+ char *applymsg;
+ char *updmsg;
initStringInfo(&err_detail);
+ if (apply_remote)
+ applymsg = "applying the remote changes.";
+ else
+ applymsg = "ignoring the remote changes.";
+
/* First, construct a detailed message describing the type of conflict */
switch (type)
{
@@ -295,13 +315,14 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
if (localts)
{
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified locally in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s."),
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by origin \"%s\" in transaction %u at %s, %s"),
get_rel_name(indexoid), origin_name,
- localxmin, timestamptz_to_str(localts));
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
/*
* The origin that modified this row has been removed. This
@@ -311,47 +332,65 @@ errdetail_apply_conflict(EState *estate, ResultRelInfo *relinfo,
* manually dropped by the user.
*/
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s."),
- get_rel_name(indexoid),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified by a non-existent origin in transaction %u at %s, %s"),
+ get_rel_name(indexoid), localxmin,
+ timestamptz_to_str(localts), applymsg);
}
else
- appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u."),
- get_rel_name(indexoid), localxmin);
+ appendStringInfo(&err_detail, _("Key already exists in unique index \"%s\", modified in transaction %u, %s"),
+ get_rel_name(indexoid), localxmin, applymsg);
break;
case CT_UPDATE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Updating the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
case CT_UPDATE_MISSING:
- appendStringInfo(&err_detail, _("Could not find the row to be updated."));
+ updmsg = "Could not find the row to be updated";
+ if (resolver == CR_APPLY_OR_SKIP && !apply_remote)
+ appendStringInfo(&err_detail, _("%s, and the UPDATE cannot be converted to an INSERT, thus skipping the remote changes."),
+ updmsg);
+ else if (resolver == CR_APPLY_OR_ERROR && !apply_remote)
+ appendStringInfo(&err_detail, _("%s, and the UPDATE cannot be converted to an INSERT, thus raising the error."),
+ updmsg);
+ else if (apply_remote)
+ appendStringInfo(&err_detail, _("%s, thus converting the UPDATE to INSERT and %s"),
+ updmsg, applymsg);
+ else
+ appendStringInfo(&err_detail, _("%s, %s"),
+ updmsg, applymsg);
break;
case CT_DELETE_ORIGIN_DIFFERS:
if (localorigin == InvalidRepOriginId)
- appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified locally in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
else if (replorigin_by_oid(localorigin, true, &origin_name))
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s."),
- origin_name, localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a different origin \"%s\" in transaction %u at %s, %s"),
+ origin_name, localxmin,
+ timestamptz_to_str(localts), applymsg);
/* The origin that modified this row has been removed. */
else
- appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s."),
- localxmin, timestamptz_to_str(localts));
+ appendStringInfo(&err_detail, _("Deleting the row that was modified by a non-existent origin in transaction %u at %s, %s"),
+ localxmin, timestamptz_to_str(localts),
+ applymsg);
break;
@@ -658,6 +697,73 @@ ValidateConflictTypeAndResolver(const char *conflict_type,
}
/*
+ * Get the conflict resolver configured at subscription level for
+ * for the given conflict type.
+ */
+static ConflictResolver
+get_conflict_resolver_internal(ConflictType type, Oid subid)
+{
+
+ ConflictResolver resolver;
+ HeapTuple tuple;
+ Datum datum;
+ char *conflict_res;
+
+ /*
+ * XXX: Currently, we fetch the conflict resolver from cache for each
+ * conflict detection. If needed, we can keep the info in global variable
+ * and fetch from cache only once after cache invalidation.
+ */
+ tuple = SearchSysCache2(SUBSCRIPTIONCONFLMAP,
+ ObjectIdGetDatum(subid),
+ CStringGetTextDatum(ConflictInfoMap[type].conflict_name));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for conflict type %u", type);
+
+ datum = SysCacheGetAttrNotNull(SUBSCRIPTIONCONFLMAP,
+ tuple, Anum_pg_subscription_conflict_confres);
+
+ conflict_res = TextDatumGetCString(datum);
+
+ for (resolver = 0; resolver < CONFLICT_NUM_RESOLVERS; resolver++)
+ {
+ if (pg_strcasecmp(ConflictResolverNames[resolver], conflict_res) == 0)
+ break;
+ }
+
+ ReleaseSysCache(tuple);
+ return resolver;
+}
+
+/*
+ * Check if a full tuple can be created from the new tuple.
+ * Return true if yes, false otherwise.
+ */
+static bool
+can_create_full_tuple(Relation localrel,
+ LogicalRepTupleData *newtup)
+{
+ int i;
+ int local_att = RelationGetNumberOfAttributes(localrel);
+
+ if (newtup->ncols != local_att)
+ return false;
+
+ /*
+ * A full tuple cannot be created if any column contains a toast value.
+ * Columns with toast values are marked as LOGICALREP_COLUMN_UNCHANGED.
+ */
+ for (i = 0; i < newtup->ncols; i++)
+ {
+ if (newtup->colstatus[i] == LOGICALREP_COLUMN_UNCHANGED)
+ return false;
+ }
+
+ return true;
+}
+
+/*
* Common 'CONFLICT RESOLVER' parsing function for CREATE and ALTER
* SUBSCRIPTION commands.
*
@@ -922,3 +1028,47 @@ RemoveSubConflictResolvers(Oid subid)
table_endscan(scan);
table_close(rel, RowExclusiveLock);
}
+
+/*
+ * Find the resolver for the given conflict type and subscription.
+ *
+ * Set 'apply_remote' to true if remote tuple should be applied,
+ * false otherwise.
+ */
+ConflictResolver
+GetConflictResolver(Oid subid, ConflictType type, Relation localrel,
+ LogicalRepTupleData *newtup, bool *apply_remote)
+{
+ ConflictResolver resolver;
+
+ resolver = get_conflict_resolver_internal(type, subid);
+
+ switch (resolver)
+ {
+ case CR_APPLY_REMOTE:
+ *apply_remote = true;
+ break;
+ case CR_APPLY_OR_SKIP:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_APPLY_OR_ERROR:
+ if (can_create_full_tuple(localrel, newtup))
+ *apply_remote = true;
+ else
+ *apply_remote = false;
+ break;
+ case CR_KEEP_LOCAL:
+ case CR_SKIP:
+ case CR_ERROR:
+ *apply_remote = false;
+ break;
+ default:
+ elog(ERROR, "Conflict %s is detected! Unrecogonized conflict resolution method",
+ ConflictInfoMap[type].conflict_name);
+ }
+
+ return resolver;
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 925dff9..abda71a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -382,7 +382,9 @@ static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void apply_handle_commit_internal(LogicalRepCommitData *commit_data);
static void apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot);
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry);
static void apply_handle_update_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
TupleTableSlot *remoteslot,
@@ -2451,10 +2453,10 @@ apply_handle_insert(StringInfo s)
/* For a partitioned table, insert the tuple into a partition. */
if (rel->localrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
apply_handle_tuple_routing(edata,
- remoteslot, NULL, CMD_INSERT);
+ remoteslot, &newtup, CMD_INSERT);
else
apply_handle_insert_internal(edata, edata->targetRelInfo,
- remoteslot);
+ remoteslot, &newtup, rel);
finish_edata(edata);
@@ -2477,9 +2479,13 @@ apply_handle_insert(StringInfo s)
static void
apply_handle_insert_internal(ApplyExecutionData *edata,
ResultRelInfo *relinfo,
- TupleTableSlot *remoteslot)
+ TupleTableSlot *remoteslot,
+ LogicalRepTupleData *newtup,
+ LogicalRepRelMapEntry *rel_entry)
{
EState *estate = edata->estate;
+ MemoryContext oldctx;
+ TupleTableSlot *conflictslot = NULL;
/* We must open indexes here. */
ExecOpenIndices(relinfo, true);
@@ -2487,7 +2493,37 @@ apply_handle_insert_internal(ApplyExecutionData *edata,
/* Do the insert. */
TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_INSERT);
- ExecSimpleRelationInsert(relinfo, estate, remoteslot);
+
+ ExecSimpleRelationInsert(relinfo, estate, remoteslot, &conflictslot,
+ MySubscription->oid);
+
+ /*
+ * If a conflict is detected, update the conflicting tuple by converting
+ * the remote INSERT to an UPDATE. Note that conflictslot will have the
+ * conflicting tuple only if the resolver is in favor of applying the
+ * changes, otherwise it will be NULL.
+ */
+ if (conflictslot)
+ {
+ EPQState epqstate;
+
+ EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
+
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, conflictslot, rel_entry, newtup);
+ MemoryContextSwitchTo(oldctx);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
+
+ /* Do the update */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, conflictslot,
+ remoteslot);
+
+ EvalPlanQualEnd(&epqstate);
+ ExecDropSingleTupleTableSlot(conflictslot);
+ }
/* Cleanup. */
ExecCloseIndices(relinfo);
@@ -2669,6 +2705,8 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TupleTableSlot *localslot;
bool found;
MemoryContext oldctx;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, true);
@@ -2690,36 +2728,48 @@ apply_handle_update_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
{
TupleTableSlot *newslot;
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_ORIGIN_DIFFERS,
+ localrel, NULL,
+ &apply_remote);
+
/* Store the new tuple for conflict reporting */
newslot = table_slot_create(localrel, &estate->es_tupleTable);
slot_store_data(newslot, relmapentry, newtup);
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot, localslot, newslot,
- InvalidOid, localxmin, localorigin, localts);
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_ORIGIN_DIFFERS, resolver,
+ remoteslot, localslot, newslot, InvalidOid,
+ localxmin, localorigin, localts, apply_remote);
}
- /* Process and store remote tuple in the slot */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, relmapentry, newtup);
- MemoryContextSwitchTo(oldctx);
+ /*
+ * Apply the change if the configured resolver is in favor of that;
+ * otherwise, ignore the remote update.
+ */
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
+ MemoryContextSwitchTo(oldctx);
- EvalPlanQualSetSlot(&epqstate, remoteslot);
+ EvalPlanQualSetSlot(&epqstate, remoteslot);
- InitConflictIndexes(relinfo);
+ InitConflictIndexes(relinfo);
- /* Do the actual update. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
- ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
- remoteslot);
+ /* Do the actual update. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_UPDATE);
+ ExecSimpleRelationUpdate(relinfo, estate, &epqstate, localslot,
+ remoteslot);
+ }
}
else
{
@@ -2729,13 +2779,29 @@ apply_handle_update_internal(ApplyExecutionData *edata,
slot_store_data(newslot, relmapentry, newtup);
/*
- * The tuple to be updated could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be updated could not be found. Report the conflict. If
+ * the configured resolver is in favor of applying the change, convert
+ * UPDATE to INSERT and apply the change.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_UPDATE_MISSING,
- remoteslot, NULL, newslot,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(MySubscription->oid, CT_UPDATE_MISSING,
+ localrel, newtup, &apply_remote);
+
+ ReportApplyConflict(estate, relinfo, CT_UPDATE_MISSING, resolver,
+ remoteslot, NULL, newslot, InvalidOid,
+ InvalidTransactionId, InvalidRepOriginId,
+ 0, apply_remote);
+
+ if (apply_remote)
+ {
+ /* Process and store remote tuple in the slot */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot, relmapentry, newtup);
+ slot_fill_defaults(relmapentry, estate, remoteslot);
+ MemoryContextSwitchTo(oldctx);
+
+ apply_handle_insert_internal(edata, relinfo, remoteslot, newtup,
+ relmapentry);
+ }
}
/* Cleanup. */
@@ -2848,6 +2914,8 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
EPQState epqstate;
TupleTableSlot *localslot;
bool found;
+ bool apply_remote = true;
+ ConflictResolver resolver;
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
ExecOpenIndices(relinfo, false);
@@ -2863,31 +2931,50 @@ apply_handle_delete_internal(ApplyExecutionData *edata,
TimestampTz localts;
/*
- * Report the conflict if the tuple was modified by a different
- * origin.
+ * Report the conflict and configured resolver if the tuple was
+ * modified by a different origin.
*/
if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
localorigin != replorigin_session_origin)
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_ORIGIN_DIFFERS,
- remoteslot, localslot, NULL,
- InvalidOid, localxmin, localorigin, localts);
+ {
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_DELETE_ORIGIN_DIFFERS,
+ localrel, NULL, &apply_remote);
+
+ ReportApplyConflict(estate, relinfo, CT_DELETE_ORIGIN_DIFFERS,
+ resolver, remoteslot, localslot, NULL,
+ InvalidOid, localxmin, localorigin, localts,
+ apply_remote);
+ }
- EvalPlanQualSetSlot(&epqstate, localslot);
+ /*
+ * Apply the change if configured resolver is in favor of that;
+ * otherwise, ignore the remote delete.
+ */
+ if (apply_remote)
+ {
+ EvalPlanQualSetSlot(&epqstate, localslot);
- /* Do the actual delete. */
- TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ /* Do the actual delete. */
+ TargetPrivilegesCheck(relinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(relinfo, estate, &epqstate, localslot);
+ }
}
else
{
/*
- * The tuple to be deleted could not be found. Do nothing except for
- * emitting a log message.
+ * The tuple to be deleted could not be found. Based on resolver
+ * configured, either skip and log a message or emit an error.
*/
- ReportApplyConflict(estate, relinfo, LOG, CT_DELETE_MISSING,
- remoteslot, NULL, NULL,
- InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ resolver = GetConflictResolver(MySubscription->oid, CT_DELETE_MISSING,
+ localrel, NULL, &apply_remote);
+
+ /* Resolver is set to skip, thus report the conflict and skip */
+ if (!apply_remote)
+ ReportApplyConflict(estate, relinfo, CT_DELETE_MISSING,
+ resolver, remoteslot, NULL, NULL,
+ InvalidOid, InvalidTransactionId,
+ InvalidRepOriginId, 0, apply_remote);
}
/* Cleanup. */
@@ -3021,19 +3108,21 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
}
MemoryContextSwitchTo(oldctx);
- /* Check if we can do the update or delete on the leaf partition. */
- if (operation == CMD_UPDATE || operation == CMD_DELETE)
+ if (operation == CMD_INSERT || operation == CMD_UPDATE ||
+ operation == CMD_DELETE)
{
- part_entry = logicalrep_partition_open(relmapentry, partrel,
- attrmap);
- check_relation_updatable(part_entry);
+ part_entry = logicalrep_partition_open(relmapentry, partrel, attrmap);
+
+ /* Check if we can do the update or delete on the leaf partition */
+ if (operation != CMD_INSERT)
+ check_relation_updatable(part_entry);
}
switch (operation)
{
case CMD_INSERT:
apply_handle_insert_internal(edata, partrelinfo,
- remoteslot_part);
+ remoteslot_part, newtup, part_entry);
break;
case CMD_DELETE:
@@ -3059,6 +3148,9 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
RepOriginId localorigin;
TransactionId localxmin;
TimestampTz localts;
+ LogicalRepRelMapEntry *part_entry_new = NULL;
+ ConflictResolver resolver;
+ bool apply_remote = true;
/* Get the matching local tuple from the partition. */
found = FindReplTupleInLocalRel(edata, partrel,
@@ -3071,47 +3163,87 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
/* Store the new tuple for conflict reporting */
slot_store_data(newslot, part_entry, newtup);
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_MISSING,
+ partrel, newtup,
+ &apply_remote);
/*
- * The tuple to be updated could not be found. Do nothing
- * except for emitting a log message.
+ * The tuple to be updated could not be found. Report the
+ * conflict and resolver. And take action based on the
+ * configured resolver.
*/
ReportApplyConflict(estate, partrelinfo,
- LOG, CT_UPDATE_MISSING,
+ CT_UPDATE_MISSING, resolver,
remoteslot_part, NULL, newslot,
InvalidOid, InvalidTransactionId,
- InvalidRepOriginId, 0);
+ InvalidRepOriginId, 0, apply_remote);
- return;
+ /*
+ * Resolver is in favour of applying the remote changes.
+ * Prepare the slot for the INSERT.
+ */
+ if (apply_remote)
+ {
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_store_data(remoteslot_part, part_entry, newtup);
+ slot_fill_defaults(part_entry, estate, remoteslot_part);
+ MemoryContextSwitchTo(oldctx);
+ }
}
-
- /*
- * Report the conflict if the tuple was modified by a
- * different origin.
- */
- if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
- localorigin != replorigin_session_origin)
+ else
{
- TupleTableSlot *newslot;
+ /*
+ * The tuple to be updated is found. Report and resolve
+ * the conflict if the tuple was modified by a different
+ * origin.
+ */
+ if (GetTupleTransactionInfo(localslot, &localxmin, &localorigin, &localts) &&
+ localorigin != replorigin_session_origin)
+ {
+ TupleTableSlot *newslot;
- /* Store the new tuple for conflict reporting */
- newslot = table_slot_create(partrel, &estate->es_tupleTable);
- slot_store_data(newslot, part_entry, newtup);
+ /* Store the new tuple for conflict reporting */
+ newslot = table_slot_create(partrel, &estate->es_tupleTable);
+ slot_store_data(newslot, part_entry, newtup);
- ReportApplyConflict(estate, partrelinfo, LOG, CT_UPDATE_ORIGIN_DIFFERS,
- remoteslot_part, localslot, newslot,
- InvalidOid, localxmin, localorigin,
- localts);
- }
+ resolver = GetConflictResolver(MySubscription->oid,
+ CT_UPDATE_ORIGIN_DIFFERS,
+ partrel, NULL,
+ &apply_remote);
- /*
- * Apply the update to the local tuple, putting the result in
- * remoteslot_part.
- */
- oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot_part, localslot, part_entry,
- newtup);
- MemoryContextSwitchTo(oldctx);
+ ReportApplyConflict(estate, partrelinfo, CT_UPDATE_ORIGIN_DIFFERS,
+ resolver, remoteslot_part, localslot,
+ newslot, InvalidOid, localxmin,
+ localorigin, localts, apply_remote);
+ }
+ if (apply_remote)
+ {
+ /*
+ * We can reach here in two cases:
+ *
+ * 1. If we found a tuple and no conflict is detected
+ *
+ * 2. If we found a tuple and conflict is detected,
+ * and the resolver is in favor of applying the change
+ *
+ * Putting the result in remoteslot_part.
+ */
+ oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
+ slot_modify_data(remoteslot_part, localslot, part_entry,
+ newtup);
+ MemoryContextSwitchTo(oldctx);
+ }
+ else
+
+ /*
+ * apply_remote can be toggled if resolver for
+ * update_origin_differs is set to skip. Ignore remote
+ * update.
+ */
+ return;
+
+ }
EvalPlanQualInit(&epqstate, estate, NULL, NIL, -1, NIL);
@@ -3123,23 +3255,52 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
ExecPartitionCheck(partrelinfo, remoteslot_part, estate,
false))
{
- /*
- * Yes, so simply UPDATE the partition. We don't call
- * apply_handle_update_internal() here, which would
- * normally do the following work, to avoid repeating some
- * work already done above to find the local tuple in the
- * partition.
- */
- ExecOpenIndices(partrelinfo, true);
- InitConflictIndexes(partrelinfo);
-
- EvalPlanQualSetSlot(&epqstate, remoteslot_part);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
- ACL_UPDATE);
- ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
- localslot, remoteslot_part);
+ if (found && apply_remote)
+ {
+ /*
+ * Yes, so simply UPDATE the partition. We don't call
+ * apply_handle_update_internal() here, which would
+ * normally do the following work, to avoid repeating
+ * some work already done above to find the local
+ * tuple in the partition.
+ *
+ * Do the update in cases - 1. found a tuple and no
+ * conflict is detected 2. update_origin_differs
+ * conflict is detected for the found tuple and the
+ * resolver is in favour of applying the update.
+ */
+ ExecOpenIndices(partrelinfo, true);
+ InitConflictIndexes(partrelinfo);
+
+ EvalPlanQualSetSlot(&epqstate, remoteslot_part);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc,
+ ACL_UPDATE);
+ ExecSimpleRelationUpdate(partrelinfo, estate, &epqstate,
+ localslot, remoteslot_part);
+ ExecCloseIndices(partrelinfo);
+ }
+ else if (apply_remote)
+ {
+ /*
+ * Tuple is not found but update_missing resolver is
+ * in favour of applying the change as INSERT.
+ */
+ apply_handle_insert_internal(edata, partrelinfo,
+ remoteslot_part, newtup,
+ part_entry);
+ }
}
- else
+
+ /*
+ * Updated tuple doesn't satisfy the current partition's
+ * constraint.
+ *
+ * Proceed by always applying the update (as 'apply_remote' is
+ * by default true). The 'apply_remote' can be OFF as well if
+ * the resolver for update_missing conflict conveys to skip
+ * the update.
+ */
+ else if (apply_remote)
{
/* Move the tuple into the new partition. */
@@ -3180,12 +3341,20 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
get_namespace_name(RelationGetNamespace(partrel_new)),
RelationGetRelationName(partrel_new));
- ExecOpenIndices(partrelinfo, false);
-
- /* DELETE old tuple found in the old partition. */
- EvalPlanQualSetSlot(&epqstate, localslot);
- TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
- ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ /*
+ * If tuple is found, delete it from old partition. We can
+ * reach this flow even for the case when the 'found' flag
+ * is false for 'update_missing' conflict and resolver is
+ * in favor of inserting the tuple.
+ */
+ if (found)
+ {
+ ExecOpenIndices(partrelinfo, false);
+ EvalPlanQualSetSlot(&epqstate, localslot);
+ TargetPrivilegesCheck(partrelinfo->ri_RelationDesc, ACL_DELETE);
+ ExecSimpleRelationDelete(partrelinfo, estate, &epqstate, localslot);
+ ExecCloseIndices(partrelinfo);
+ }
/* INSERT new tuple into the new partition. */
@@ -3201,22 +3370,27 @@ apply_handle_tuple_routing(ApplyExecutionData *edata,
map = ExecGetRootToChildMap(partrelinfo_new, estate);
if (map != NULL)
{
+ attrmap = map->attrMap;
remoteslot_part = execute_attr_map_slot(map->attrMap,
remoteslot,
remoteslot_part);
}
else
{
+ attrmap = NULL;
remoteslot_part = ExecCopySlot(remoteslot_part,
remoteslot);
slot_getallattrs(remoteslot);
}
MemoryContextSwitchTo(oldctx);
+ part_entry_new = logicalrep_partition_open(part_entry,
+ partrel_new,
+ attrmap);
apply_handle_insert_internal(edata, partrelinfo_new,
- remoteslot_part);
+ remoteslot_part, newtup,
+ part_entry_new);
}
- ExecCloseIndices(partrelinfo);
EvalPlanQualEnd(&epqstate);
}
break;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 69c3ebf..f390975 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -18,6 +18,8 @@
#include "fmgr.h"
#include "nodes/lockoptions.h"
#include "nodes/parsenodes.h"
+#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/memutils.h"
@@ -667,7 +669,8 @@ extern bool RelationFindReplTupleSeq(Relation rel, LockTupleMode lockmode,
TupleTableSlot *searchslot, TupleTableSlot *outslot);
extern void ExecSimpleRelationInsert(ResultRelInfo *resultRelInfo,
- EState *estate, TupleTableSlot *slot);
+ EState *estate, TupleTableSlot *slot,
+ TupleTableSlot **conflictsloty, Oid subid);
extern void ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
EState *estate, EPQState *epqstate,
TupleTableSlot *searchslot, TupleTableSlot *slot);
diff --git a/src/include/replication/conflict.h b/src/include/replication/conflict.h
index c5865a1..0680ef1 100644
--- a/src/include/replication/conflict.h
+++ b/src/include/replication/conflict.h
@@ -11,6 +11,7 @@
#include "nodes/execnodes.h"
#include "parser/parse_node.h"
+#include "replication/logicalrelation.h"
#include "utils/timestamp.h"
/*
@@ -91,12 +92,14 @@ extern bool GetTupleTransactionInfo(TupleTableSlot *localslot,
RepOriginId *localorigin,
TimestampTz *localts);
extern void ReportApplyConflict(EState *estate, ResultRelInfo *relinfo,
- int elevel, ConflictType type,
+ ConflictType type,
+ ConflictResolver resolver,
TupleTableSlot *searchslot,
TupleTableSlot *localslot,
TupleTableSlot *remoteslot,
Oid indexoid, TransactionId localxmin,
- RepOriginId localorigin, TimestampTz localts);
+ RepOriginId localorigin, TimestampTz localts,
+ bool apply_remote);
extern void InitConflictIndexes(ResultRelInfo *relInfo);
extern void SetSubConflictResolvers(Oid subId, List *resolvers);
extern void RemoveSubConflictResolvers(Oid confid);
@@ -109,5 +112,8 @@ extern ConflictType ValidateConflictTypeAndResolver(const char *conflict_type,
const char *conflict_resolver);
extern List *GetDefaultConflictResolvers(void);
extern void ResetSubConflictResolver(Oid subid, char *conflict_type);
-
+extern ConflictResolver GetConflictResolver(Oid subid, ConflictType type,
+ Relation localrel,
+ LogicalRepTupleData *newtup,
+ bool *apply_remote);
#endif
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index c591cd7..00ade29 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -40,6 +40,7 @@ tests += {
't/031_column_list.pl',
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
+ 't/034_conflict_resolver.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/034_conflict_resolver.pl b/src/test/subscription/t/034_conflict_resolver.pl
new file mode 100644
index 0000000..86e3ad1
--- /dev/null
+++ b/src/test/subscription/t/034_conflict_resolver.pl
@@ -0,0 +1,877 @@
+
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test the conflict detection and resolution in logical replication
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf',
+ qq(max_prepared_transactions = 10));
+$node_publisher->start;
+
+# Create subscriber node with track_commit_timestamp enabled
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+# Create table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text, comments text);
+ ALTER TABLE conf_tab ALTER COLUMN comments SET STORAGE EXTERNAL;");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT conftype, confres FROM pg_subscription_conflict ORDER BY conftype"
+);
+is( $result, qq(delete_missing|skip
+delete_origin_differs|apply_remote
+insert_exists|error
+update_exists|error
+update_missing|skip
+update_origin_differs|apply_remote),
+ "confirm that the default conflict resolvers are in place");
+
+############################################
+# Test 'apply_remote' for 'insert_exists'
+############################################
+
+# Change CONFLICT RESOLVER of insert_exists to apply_remote
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'apply_remote');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'fromsub')");
+
+# Create conflicting data on the publisher
+my $log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that remote insert is converted to an update and the remote data is updated.
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=1);");
+
+is($result, 'frompub', "remote data is kept");
+
+
+########################################
+# Test 'keep_local' for 'insert_exists'
+########################################
+
+# Change CONFLICT RESOLVER of insert_exists to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'keep_local');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'fromsub')");
+
+# Confirm that row is updated
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data 2 from local is inserted");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (2,'frompub')");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that remote insert is ignored and the local row is kept
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data FROM conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "data from local is kept");
+
+###################################
+# Test 'error' for 'insert_exists'
+###################################
+
+# Change CONFLICT RESOLVER of insert_exists to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (insert_exists = 'error');"
+);
+
+# Create local data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'fromsub')");
+
+# Create conflicting data on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (3,'frompub')");
+
+$log_offset = -s $node_subscriber->logfile;
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=insert_exists, resolution=error/,
+ $log_offset);
+
+# Truncate table on subscriber to get rid of the error
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+###################################
+# Test 'skip' for 'delete_missing'
+###################################
+
+# Delete row on publisher that is not present on the subscriber and confirm that it is skipped
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Confirm that the missing row is skipped because 'delete_missing' is set to 'skip'
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_missing, resolution=skip/,
+ $log_offset);
+
+####################################
+# Test 'error' for 'delete_missing'
+####################################
+
+# Change CONFLICT RESOLVER of delete_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error');"
+);
+
+# Capture the log offset before performing the delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+
+# Perform the delete on the publisher
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+# Confirm that this causes an error on the subscriber
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate table on subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+
+#################################################
+# Test 'apply_remote' for 'delete_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting delete on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote delete the local updated row
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, '', "delete from remote is applied");
+
+###############################################
+# Test 'keep_local' for 'delete_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'delete_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of delete_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=1);");
+
+is($result, 'frompubnew', "update from remote is kept");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+# Test the apply part
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=3);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+# Test the skip part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(4,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=4);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=4);");
+
+$node_subscriber->wait_for_log(
+ qr/DETAIL: Could not find the row to be updated, and the UPDATE cannot be converted to an INSERT, thus skipping the remote changes./,
+ $log_offset);
+
+###########################################
+# Test 'apply_or_error' for 'update_missing'
+###########################################
+
+# Test the apply part
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'apply_or_error');"
+);
+
+# Create new row on the publisher
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (5,'frompub');");
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=5);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=5);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_error/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=5);");
+
+is($result, 'frompubnew', "update from remote is converted to insert");
+
+# Test the error part
+
+# Create new row on publisher with toast data
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab (a, data, comments) VALUES(6,'frompub',repeat('abcdefghij', 200));"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=6);");
+
+# Update the row on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=6);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=apply_or_error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab;");
+
+# Create the subscription
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (2,'frompub');
+ INSERT INTO conf_tab(a, data) VALUES (3,'frompub');");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres', "DELETE FROM conf_tab WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab WHERE (a=2);");
+
+is($result, '', "update from remote is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION tap_sub;");
+
+#################################################
+# Partition table tests for UPDATE conflicts
+#################################################
+
+# Create partitioned table on publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);"
+);
+
+# Create similar table on subscriber but with partitions
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab_part (a int not null, b int not null, data text) partition by range (b);
+ ALTER TABLE conf_tab_part ADD CONSTRAINT conf_tab_part_pk primary key (a,b);
+ CREATE TABLE conf_tab_part_1 PARTITION OF conf_tab_part FOR VALUES FROM (MINVALUE) TO (100);
+ CREATE TABLE conf_tab_part_2 PARTITION OF conf_tab_part FOR VALUES FROM (101) TO (MAXVALUE);"
+);
+
+# Setup logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub_part FOR TABLE conf_tab_part with (publish_via_partition_root=true);"
+);
+
+# Create the subscription
+$appname = 'sub_part';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+#################################################
+# Test 'apply_remote' for 'update_origin_differs'
+#################################################
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=1);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is applied to the first partition");
+
+# Create a conflicting update which also changes the partition
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=1);");
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=101, data='frompubnew_p2' WHERE (a=1);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=apply_remote/,
+ $log_offset);
+
+# Confirm that the remote update overrides the local update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=1);");
+
+is($result, qq(101|frompubnew_p2),
+ "update from remote is the second partition");
+
+###############################################
+# Test 'keep_local' for 'update_origin_differs'
+###############################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to keep_local
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'keep_local');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=2);");
+
+# Create a conflicting update on the same partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, 'fromsub', "update from local is kept on the first partition");
+
+# Create a conflicting update which also changes the partition
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=keep_local/,
+ $log_offset);
+
+# Confirm that the local data is untouched by the remote update
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b, data from conf_tab_part WHERE (a=2);");
+
+is($result, qq(1|fromsub),
+ "update from local is kept on the first partition");
+
+##########################################
+# Test 'error' for 'update_origin_differs'
+##########################################
+
+# Change CONFLICT RESOLVER of update_origin_differs to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_origin_differs = 'error');"
+);
+
+# Modify data on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'fromsub' WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnew' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_origin_differs, resolution=error/,
+ $log_offset);
+
+# Drop the subscriber to remove error
+$node_subscriber->safe_psql('postgres', "DROP SUBSCRIPTION sub_part;");
+
+# Truncate the table on the publisher
+$node_publisher->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+# Truncate the table on the subscriber
+$node_subscriber->safe_psql('postgres', "TRUNCATE conf_tab_part;");
+
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION sub_part
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION pub_part;");
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+# Insert data in the publisher
+$node_publisher->safe_psql(
+ 'postgres',
+ "INSERT INTO conf_tab_part VALUES (1,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (2,1,'frompub');
+ INSERT INTO conf_tab_part VALUES (3,1,'frompub');");
+
+###########################################
+# Test 'apply_or_skip' for 'update_missing'
+###########################################
+
+# Change CONFLICT RESOLVER of update_missing to apply_or_skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'apply_or_skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=3);");
+
+is($result, 'frompubnew_p1',
+ "update from remote is converted to insert in the first partition");
+
+# Test the update which also changes the partition
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=3);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=103, data='frompubnew_p2' WHERE (a=3);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=apply_or_skip/,
+ $log_offset);
+
+# Confirm that the remote update is converted to an insert and new row applied
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT b,data from conf_tab_part WHERE (a=3);");
+
+is($result, '103|frompubnew_p2',
+ "update from remote is converted to insert in the second partition");
+
+###################################
+# Test 'skip' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to skip
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'skip');"
+);
+
+# Delete the row on the subscriber
+$node_subscriber->safe_psql('postgres',
+ "DELETE FROM conf_tab_part WHERE (a=2);");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data='frompubnew_p1' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on first partition is skipped on the subscriber");
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET b=102, data='frompubnew_p2' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/LOG: conflict detected on relation \"public.conf_tab_part_1\": conflict=update_missing, resolution=skip/,
+ $log_offset);
+
+# Confirm that the update does not change anything on the subscriber
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT data from conf_tab_part WHERE (a=2);");
+
+is($result, '',
+ "update from remote on second partition is skipped on the subscriber");
+
+###################################
+# Test 'error' for 'update_missing'
+###################################
+
+# Change CONFLICT RESOLVER of update_missing to error
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION sub_part CONFLICT RESOLVER (update_missing = 'error');"
+);
+
+# Create a conflicting update on the publisher
+$log_offset = -s $node_subscriber->logfile;
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab_part SET data = 'frompubnewer' WHERE (a=2);");
+
+$node_subscriber->wait_for_log(
+ qr/ERROR: conflict detected on relation \"public.conf_tab_part_2\": conflict=update_missing, resolution=error/,
+ $log_offset);
+
+done_testing();
--
1.8.3.1
Hello!
Sorry for being noisy, just for the case, want to notice that [1] needs
to be addressed before any real usage of conflict resolution.
[1]:
/messages/by-id/OS0PR01MB5716E30952F542E256DD72E294802@OS0PR01MB5716.jpnprd01.prod.outlook.com
I created two tests to reproduce the issue using the new conflict
resolution mechanics (based on the v16 patch).
One test detects an invalid delete_missing instead of
delete_origin_differs. Another test detects an invalid update_missing
instead of update_origin_differs.
You can change my $simulate_race_condition = 1; to 0 to make them pass.
[2]: /messages/by-id/CANtu0ohUB9ky45iiMAYN1fGyt82+cg=+UYBom=P7drb+=97G9w@mail.gmail.com
in [3]/messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com.
Best regards,
Mikhail
[2]: /messages/by-id/CANtu0ohUB9ky45iiMAYN1fGyt82+cg=+UYBom=P7drb+=97G9w@mail.gmail.com
/messages/by-id/CANtu0ohUB9ky45iiMAYN1fGyt82+cg=+UYBom=P7drb+=97G9w@mail.gmail.com
[3]: /messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com
/messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com
Attachments:
v16-0001-TAP-tests-to-reproduce-issue-with-DirtySnapshot-.patchtext/x-patch; charset=US-ASCII; name=v16-0001-TAP-tests-to-reproduce-issue-with-DirtySnapshot-.patchDownload
From 39497157ac7a5572a698b500001b9b6e2fbf69b4 Mon Sep 17 00:00:00 2001
From: nkey <nkey@toloka.ai>
Date: Mon, 21 Oct 2024 02:05:33 +0200
Subject: [PATCH v16] TAP tests to reproduce issue with DirtySnapshot scan [1]
with new conflict resolution system.
[1]: https://www.postgresql.org/message-id/flat/CANtu0oiziTBM8%2BWDtkktMZv0rhGBroYGWwqSQW%2BMzOWpmk-XEw%40mail.gmail.com#74f5f05594bb6f10b1d882a1ebce377c
---
src/backend/access/index/indexam.c | 11 ++
src/test/subscription/meson.build | 7 +-
.../subscription/t/035_delete_missing_race.pl | 138 +++++++++++++++++
.../subscription/t/036_update_missing_race.pl | 139 ++++++++++++++++++
4 files changed, 294 insertions(+), 1 deletion(-)
create mode 100644 src/test/subscription/t/035_delete_missing_race.pl
create mode 100644 src/test/subscription/t/036_update_missing_race.pl
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 1859be614c..ff7cad47c7 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -57,6 +57,8 @@
#include "utils/ruleutils.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
+#include "utils/injection_point.h"
+#include "replication/worker_internal.h"
/* ----------------------------------------------------------------
@@ -696,6 +698,15 @@ index_getnext_slot(IndexScanDesc scan, ScanDirection direction, TupleTableSlot *
* the index.
*/
Assert(ItemPointerIsValid(&scan->xs_heaptid));
+#ifdef USE_INJECTION_POINTS
+ if (!IsCatalogRelationOid(scan->indexRelation->rd_id)
+ && scan->xs_snapshot->snapshot_type == SNAPSHOT_DIRTY
+ && MySubscription)
+ {
+ INJECTION_POINT("index_getnext_slot_before_fetch_apply");
+ }
+#endif
+
if (index_fetch_heap(scan, slot))
return true;
}
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index 00ade29b02..aa7e2bd406 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -5,7 +5,10 @@ tests += {
'sd': meson.current_source_dir(),
'bd': meson.current_build_dir(),
'tap': {
- 'env': {'with_icu': icu.found() ? 'yes' : 'no'},
+ 'env': {
+ 'with_icu': icu.found() ? 'yes' : 'no',
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no'
+ },
'tests': [
't/001_rep_changes.pl',
't/002_types.pl',
@@ -41,6 +44,8 @@ tests += {
't/032_subscribe_use_index.pl',
't/033_run_as_table_owner.pl',
't/034_conflict_resolver.pl',
+ 't/035_delete_missing_race.pl',
+ 't/036_update_missing_race.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/035_delete_missing_race.pl b/src/test/subscription/t/035_delete_missing_race.pl
new file mode 100644
index 0000000000..d7cba0fa0c
--- /dev/null
+++ b/src/test/subscription/t/035_delete_missing_race.pl
@@ -0,0 +1,138 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test the conflict detection and resolution in logical replication
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+############################## Set it to 0 to make set success
+my $simulate_race_condition = 1;
+##############################
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+
+# Create subscriber node with track_commit_timestamp enabled
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node_publisher->check_extension('injection_points') || !$node_subscriber->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+# Create table on publisher with additional index to disable HOT updates
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text);
+ CREATE INDEX data_index ON conf_tab(data);");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text);
+ CREATE INDEX data_index ON conf_tab(data);");
+
+# Set up extension to simulate race condition
+$node_subscriber->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+$node_publisher->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Insert row to be updated later
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (disable_on_error = true);"); # mark subscription as disable_on_error to keep test simple
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+############################################
+# Race condition because of DirtySnapshot
+############################################
+
+# Setup CONFLICT RESOLVER
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (delete_missing = 'error', delete_origin_differs = 'apply_remote');"
+);
+
+my $psql_session_subscriber = $node_subscriber->background_psql('postgres');
+if ($simulate_race_condition)
+{
+ $node_subscriber->safe_psql('postgres', "SELECT injection_points_attach('index_getnext_slot_before_fetch_apply', 'wait')");
+}
+
+my $log_offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres', "DELETE FROM conf_tab WHERE a=1;");
+
+
+if ($simulate_race_condition)
+{
+ $node_subscriber->wait_for_event('logical replication apply worker', 'index_getnext_slot_before_fetch_apply');
+}
+
+$psql_session_subscriber->query_until(
+ qr/start/, qq[
+ \\echo start
+ UPDATE conf_tab SET data = 'fromsubnew' WHERE (a=1);
+]);
+
+
+if ($simulate_race_condition)
+{
+ $node_subscriber->safe_psql('postgres',"
+ SELECT injection_points_wakeup('index_getnext_slot_before_fetch_apply');
+ SELECT injection_points_detach('index_getnext_slot_before_fetch_apply');
+ ");
+}
+
+$node_subscriber->wait_for_log(
+ qr/conflict detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+ok(!$node_subscriber->log_contains(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=delete_missing, resolution=error/,
+ $log_offset), 'invalid conflict detected');
+
+ok($node_subscriber->log_contains(
+ qr/LOG: conflict detected on relation "public.conf_tab": conflict=delete_origin_differs, resolution=apply_remote/,
+ $log_offset), 'correct conflict detected');
+
+ok(!$node_subscriber->log_contains(
+ qr/LOG: subscription \"tap_sub\" has been disabled because of an error/,
+ $log_offset), 'subscription is disabled');
+
+done_testing();
diff --git a/src/test/subscription/t/036_update_missing_race.pl b/src/test/subscription/t/036_update_missing_race.pl
new file mode 100644
index 0000000000..6316c23a78
--- /dev/null
+++ b/src/test/subscription/t/036_update_missing_race.pl
@@ -0,0 +1,139 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test the conflict detection and resolution in logical replication
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+############################## Set it to 0 to make set success
+my $simulate_race_condition = 1;
+##############################
+
+###############################
+# Setup
+###############################
+
+# Initialize publisher node
+my $node_publisher = PostgreSQL::Test::Cluster->new('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+
+# Create subscriber node with track_commit_timestamp enabled
+my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$node_subscriber->init;
+$node_subscriber->append_conf('postgresql.conf',
+ qq(track_commit_timestamp = on));
+$node_subscriber->start;
+
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node_publisher->check_extension('injection_points') || !$node_subscriber->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+# Create table on publisher with additional index to disable HOT updates
+$node_publisher->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text);
+ CREATE INDEX data_index ON conf_tab(data);");
+
+# Create similar table on subscriber
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE TABLE conf_tab(a int PRIMARY key, data text);
+ CREATE INDEX data_index ON conf_tab(data);");
+
+# Set up extension to simulate race condition
+$node_subscriber->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+$node_publisher->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tap_pub FOR TABLE conf_tab");
+
+# Insert row to be updated later
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO conf_tab(a, data) VALUES (1,'frompub')");
+
+# Create the subscription
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql(
+ 'postgres',
+ "CREATE SUBSCRIPTION tap_sub
+ CONNECTION '$publisher_connstr application_name=$appname'
+ PUBLICATION tap_pub
+ WITH (disable_on_error = true);"); # mark subscription as disable_on_error to keep test simple
+
+# Wait for initial table sync to finish
+$node_subscriber->wait_for_subscription_sync($node_publisher, $appname);
+
+############################################
+# Race condition because of DirtySnapshot
+############################################
+
+# Setup CONFLICT RESOLVER
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tap_sub CONFLICT RESOLVER (update_origin_differs = 'apply_remote', update_missing = 'error');"
+);
+
+my $psql_session_subscriber = $node_subscriber->background_psql('postgres');
+if ($simulate_race_condition)
+{
+ $node_subscriber->safe_psql('postgres', "SELECT injection_points_attach('index_getnext_slot_before_fetch_apply', 'wait')");
+}
+
+my $log_offset = -s $node_subscriber->logfile;
+
+$node_publisher->safe_psql('postgres',
+ "UPDATE conf_tab SET data = 'frompubnew' WHERE (a=1);");
+
+
+if ($simulate_race_condition)
+{
+ $node_subscriber->wait_for_event('logical replication apply worker', 'index_getnext_slot_before_fetch_apply');
+}
+
+$psql_session_subscriber->query_until(
+ qr/start/, qq[
+ \\echo start
+ UPDATE conf_tab SET data = 'fromsubnew' WHERE (a=1);
+]);
+
+
+if ($simulate_race_condition)
+{
+ $node_subscriber->safe_psql('postgres',"
+ SELECT injection_points_wakeup('index_getnext_slot_before_fetch_apply');
+ SELECT injection_points_detach('index_getnext_slot_before_fetch_apply');
+ ");
+}
+
+$node_subscriber->wait_for_log(
+ qr/conflict detected on relation \"public.conf_tab\"/,
+ $log_offset);
+
+ok(!$node_subscriber->log_contains(
+ qr/ERROR: conflict detected on relation \"public.conf_tab\": conflict=update_missing, resolution=error/,
+ $log_offset), 'invalid conflict detected');
+
+ok($node_subscriber->log_contains(
+ qr/LOG: conflict detected on relation "public.conf_tab": conflict=update_origin_differs, resolution=apply_remote./,
+ $log_offset), 'correct conflict detected');
+
+ok(!$node_subscriber->log_contains(
+ qr/LOG: subscription \"tap_sub\" has been disabled because of an error/,
+ $log_offset), 'subscription is disabled');
+
+done_testing();
--
2.43.0
On Fri, Oct 18, 2024 at 4:30 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:
On Wednesday, October 9, 2024 2:34 PM shveta malik <shveta.malik@gmail.com> wrote:
On Wed, Oct 9, 2024 at 8:58 AM shveta malik <shveta.malik@gmail.com>
wrote:On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond
<nisha.moond412@gmail.com> wrote:
Please find few comments on v14-patch004:
patch004:
1)
GetConflictResolver currently errors out when the resolver is last_update_wins
and track_commit_timestamp is disabled. It means every conflict resolution
with this resolver will keep on erroring out. I am not sure if we should emit
ERROR here. We do emit ERROR when someone tries to configure
last_update_wins but track_commit_timestamp is disabled. I think that should
suffice. The one in GetConflictResolver can be converted to WARNING max.What could be the side-effect if we do not emit error here? In such a case, the
local timestamp will be 0 and remote change will always win.
Is that right? If so, then if needed, we can emit a warning saying something like:
'track_commit_timestamp is disabled and thus remote change is applied
always.'Thoughts?
I think simply reporting a warning and applying remote changes without further
action could lead to data inconsistencies between nodes. Considering the
potential challenges and time required to recover from these inconsistencies, I
prefer to keep reporting errors, in which case users have an opportunity to
resolve the issue by enabling track_commit_timestamp.
Okay, makes sense. We should raise ERROR then.
thanks
Shveta
Hello hackers,
Hey I'm Diego and I do work for Percona and started to work on PostgreSQL
and I would like to contribute to the project moving forward.
I have been following this thread since the beginning, but due to my
limited knowledge of the overall code structure, my first review of the
provided patches was more focused on validating the logic and general flow.
I have been testing the provided patches and so far the only issue I have
is the one reported about DirtySnapshot scans over a B-tree with parallel
updates, which may skip/not find some records.
That said, I'd like to know if it's worthwhile pulling the proposed fix on
[0]: /messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com
better solutions being discussed?
Thanks for your attention,
Diego
[0]: /messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com
/messages/by-id/CANtu0oiziTBM8+WDtkktMZv0rhGBroYGWwqSQW+MzOWpmk-XEw@mail.gmail.com
On Mon, Oct 21, 2024 at 2:04 AM shveta malik <shveta.malik@gmail.com> wrote:
Show quoted text
On Fri, Oct 18, 2024 at 4:30 PM Zhijie Hou (Fujitsu)
<houzj.fnst@fujitsu.com> wrote:On Wednesday, October 9, 2024 2:34 PM shveta malik <
shveta.malik@gmail.com> wrote:
On Wed, Oct 9, 2024 at 8:58 AM shveta malik <shveta.malik@gmail.com>
wrote:On Tue, Oct 8, 2024 at 3:12 PM Nisha Moond
<nisha.moond412@gmail.com> wrote:
Please find few comments on v14-patch004:
patch004:
1)
GetConflictResolver currently errors out when the resolver islast_update_wins
and track_commit_timestamp is disabled. It means every conflict
resolution
with this resolver will keep on erroring out. I am not sure if we
should emit
ERROR here. We do emit ERROR when someone tries to configure
last_update_wins but track_commit_timestamp is disabled. I think thatshould
suffice. The one in GetConflictResolver can be converted to WARNING
max.
What could be the side-effect if we do not emit error here? In such a
case, the
local timestamp will be 0 and remote change will always win.
Is that right? If so, then if needed, we can emit a warning sayingsomething like:
'track_commit_timestamp is disabled and thus remote change is applied
always.'Thoughts?
I think simply reporting a warning and applying remote changes without
further
action could lead to data inconsistencies between nodes. Considering the
potential challenges and time required to recover from theseinconsistencies, I
prefer to keep reporting errors, in which case users have an opportunity
to
resolve the issue by enabling track_commit_timestamp.
Okay, makes sense. We should raise ERROR then.
thanks
Shveta
Hello, everyone!
Just curious - are any plans to continue to work on this project?
Best regards,
Mikhail.
On Mon, Jan 13, 2025 at 11:01 PM Michail Nikolaev
<michail.nikolaev@gmail.com> wrote:
Just curious - are any plans to continue to work on this project?
Yes, but our intention is to reach a consensus/conclusion about how to
deal with update_delete conflicts. The topic is discussed in the CF
entry [1]https://commitfest.postgresql.org/51/5378/.
If you want to help move forward with this work, it is better to help
with the review of work in the CF entry [1]https://commitfest.postgresql.org/51/5378/.
[1]: https://commitfest.postgresql.org/51/5378/
--
With Regards,
Amit Kapila.