pgsql: Logical replication support for initial data copy
Logical replication support for initial data copy
Add functionality for a new subscription to copy the initial data in the
tables and then sync with the ongoing apply process.
For the copying, add a new internal COPY option to have the COPY source
data provided by a callback function. The initial data copy works on
the subscriber by receiving COPY data from the publisher and then
providing it locally into a COPY that writes to the destination table.
A WAL receiver can now execute full SQL commands. This is used here to
obtain information about tables and publications.
Several new options were added to CREATE and ALTER SUBSCRIPTION to
control whether and when initial table syncing happens.
Change pg_dump option --no-create-subscription-slots to
--no-subscription-connect and use the new CREATE SUBSCRIPTION
... NOCONNECT option for that.
Author: Petr Jelinek <petr.jelinek@2ndquadrant.com>
Tested-by: Erik Rijkers <er@xs4all.nl>
Branch
------
master
Details
-------
http://git.postgresql.org/pg/commitdiff/7c4f52409a8c7d85ed169bbbc1f6092274d03920
Modified Files
--------------
contrib/file_fdw/file_fdw.c | 5 +-
doc/src/sgml/catalogs.sgml | 78 ++
doc/src/sgml/config.sgml | 25 +
doc/src/sgml/logical-replication.sgml | 55 +-
doc/src/sgml/monitoring.sgml | 9 +-
doc/src/sgml/protocol.sgml | 9 +-
doc/src/sgml/ref/alter_subscription.sgml | 50 +-
doc/src/sgml/ref/create_subscription.sgml | 38 +
doc/src/sgml/ref/pg_dump.sgml | 15 +-
src/backend/catalog/Makefile | 1 +
src/backend/catalog/heap.c | 6 +
src/backend/catalog/pg_publication.c | 4 +-
src/backend/catalog/pg_subscription.c | 282 +++++++
src/backend/catalog/system_views.sql | 1 +
src/backend/commands/copy.c | 23 +-
src/backend/commands/subscriptioncmds.c | 468 ++++++++++--
src/backend/parser/gram.y | 39 +-
src/backend/postmaster/pgstat.c | 6 +
.../libpqwalreceiver/libpqwalreceiver.c | 203 ++++-
src/backend/replication/logical/Makefile | 2 +-
src/backend/replication/logical/launcher.c | 130 +++-
src/backend/replication/logical/relation.c | 7 +
src/backend/replication/logical/snapbuild.c | 85 ++-
src/backend/replication/logical/tablesync.c | 840 +++++++++++++++++++++
src/backend/replication/logical/worker.c | 203 +++--
src/backend/replication/repl_gram.y | 32 +-
src/backend/replication/repl_scanner.l | 5 +-
src/backend/replication/walsender.c | 104 ++-
src/backend/tcop/postgres.c | 5 +-
src/backend/utils/adt/misc.c | 20 +
src/backend/utils/cache/syscache.c | 14 +-
src/backend/utils/misc/guc.c | 12 +
src/bin/pg_dump/pg_backup.h | 2 +-
src/bin/pg_dump/pg_dump.c | 9 +-
src/bin/pg_dump/t/002_pg_dump.pl | 2 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/indexing.h | 7 +-
src/include/catalog/pg_proc.h | 5 +-
src/include/catalog/pg_subscription_rel.h | 78 ++
src/include/commands/copy.h | 5 +-
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 13 +
src/include/nodes/replnodes.h | 9 +
src/include/parser/kwlist.h | 1 +
src/include/pgstat.h | 4 +-
src/include/replication/logical.h | 13 +-
src/include/replication/logicallauncher.h | 1 +
src/include/replication/snapbuild.h | 1 +
src/include/replication/walreceiver.h | 67 +-
src/include/replication/walsender.h | 12 +-
src/include/replication/worker_internal.h | 30 +-
src/include/utils/syscache.h | 1 +
src/test/regress/expected/object_address.out | 3 +-
src/test/regress/expected/rules.out | 3 +-
src/test/regress/expected/sanity_check.out | 1 +
src/test/regress/expected/subscription.out | 45 +-
src/test/regress/sql/object_address.sql | 2 +-
src/test/regress/sql/subscription.sql | 11 +-
src/test/subscription/t/001_rep_changes.pl | 36 +-
src/test/subscription/t/002_types.pl | 6 +
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 159 ++++
62 files changed, 2966 insertions(+), 341 deletions(-)
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.
Still analyzing the rest.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
logical-replication-copy-patch-fixes.patchtext/x-patch; name=logical-replication-copy-patch-fixes.patchDownload
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 67c7de6..93d4eb2 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4450,7 +4450,10 @@ _copyAlterSubscriptionStmt(const AlterSubscriptionStmt *from)
{
AlterSubscriptionStmt *newnode = makeNode(AlterSubscriptionStmt);
+ COPY_SCALAR_FIELD(kind);
COPY_STRING_FIELD(subname);
+ COPY_STRING_FIELD(conninfo);
+ COPY_NODE_FIELD(publication);
COPY_NODE_FIELD(options);
return newnode;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4d8e7fe..6b40b56 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2199,7 +2199,10 @@ static bool
_equalAlterSubscriptionStmt(const AlterSubscriptionStmt *a,
const AlterSubscriptionStmt *b)
{
+ COMPARE_SCALAR_FIELD(kind);
COMPARE_STRING_FIELD(subname);
+ COMPARE_STRING_FIELD(conninfo);
+ COMPARE_NODE_FIELD(publication);
COMPARE_NODE_FIELD(options);
return true;
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index bf96d34..5bebca3 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -46,7 +46,7 @@ typedef struct LogicalRepWorker
} LogicalRepWorker;
/* Memory context for cached variables in apply worker. */
-MemoryContext ApplyCacheContext;
+extern MemoryContext ApplyCacheContext;
/* libpqreceiver connection */
extern struct WalReceiverConn *wrconn;
Petr,
* Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote:
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.
Pretty sure I saw a complaint about a trailing comma in a struct or enum
too..
Thanks!
Stephen
On 23/03/17 15:17, Stephen Frost wrote:
Petr,
* Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote:
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.Pretty sure I saw a complaint about a trailing comma in a struct or enum
too..
Ah, missed that in the wall of text, thanks, fixed that too.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
logical-replication-copy-patch-fixesv2.patchtext/x-patch; name=logical-replication-copy-patch-fixesv2.patchDownload
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 67c7de6..93d4eb2 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4450,7 +4450,10 @@ _copyAlterSubscriptionStmt(const AlterSubscriptionStmt *from)
{
AlterSubscriptionStmt *newnode = makeNode(AlterSubscriptionStmt);
+ COPY_SCALAR_FIELD(kind);
COPY_STRING_FIELD(subname);
+ COPY_STRING_FIELD(conninfo);
+ COPY_NODE_FIELD(publication);
COPY_NODE_FIELD(options);
return newnode;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 4d8e7fe..6b40b56 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2199,7 +2199,10 @@ static bool
_equalAlterSubscriptionStmt(const AlterSubscriptionStmt *a,
const AlterSubscriptionStmt *b)
{
+ COMPARE_SCALAR_FIELD(kind);
COMPARE_STRING_FIELD(subname);
+ COMPARE_STRING_FIELD(conninfo);
+ COMPARE_NODE_FIELD(publication);
COMPARE_NODE_FIELD(options);
return true;
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index fb55c30..cc31bc6 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -175,7 +175,7 @@ typedef enum
WALRCV_OK_TUPLES, /* Query returned tuples. */
WALRCV_OK_COPY_IN, /* Query started COPY FROM. */
WALRCV_OK_COPY_OUT, /* Query started COPY TO. */
- WALRCV_OK_COPY_BOTH, /* Query started COPY BOTH replication protocol. */
+ WALRCV_OK_COPY_BOTH /* Query started COPY BOTH replication protocol. */
} WalRcvExecStatus;
/*
diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h
index bf96d34..5bebca3 100644
--- a/src/include/replication/worker_internal.h
+++ b/src/include/replication/worker_internal.h
@@ -46,7 +46,7 @@ typedef struct LogicalRepWorker
} LogicalRepWorker;
/* Memory context for cached variables in apply worker. */
-MemoryContext ApplyCacheContext;
+extern MemoryContext ApplyCacheContext;
/* libpqreceiver connection */
extern struct WalReceiverConn *wrconn;
On 3/23/17 10:12, Petr Jelinek wrote:
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.
Fixed.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
On 3/23/17 10:25, Petr Jelinek wrote:
On 23/03/17 15:17, Stephen Frost wrote:
Petr,
* Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote:
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.Pretty sure I saw a complaint about a trailing comma in a struct or enum
too..Ah, missed that in the wall of text, thanks, fixed that too.
Also fixed.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers
On Fri, Mar 24, 2017 at 12:59 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
On 3/23/17 10:25, Petr Jelinek wrote:
On 23/03/17 15:17, Stephen Frost wrote:
Petr,
* Petr Jelinek (petr.jelinek@2ndquadrant.com) wrote:
On 23/03/17 14:47, Petr Jelinek wrote:
I am looking into buildfarm failures.
Looks like the clang issue (mylodon, longfin) is because we are missing
extern in include/replication/worker_internal.h for ApplyCacheContext.Still analyzing the rest.
Ah missing fields in copy/equal function for AlterSubscriptionStmt.
Attached patch adds the above mentioned extern and fixes the copy and
equal functions.Pretty sure I saw a complaint about a trailing comma in a struct or enum
too..Ah, missed that in the wall of text, thanks, fixed that too.
Also fixed.
The copyright in some files(tablesync.c and pg_subscriptiion_rel.h)
introduced by this commit are old. Is that intentional? Attached
patch.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
Attachments:
update_copyright.patchapplication/octet-stream; name=update_copyright.patchDownload
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 3e16b0d..a067fe3 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -2,7 +2,7 @@
* tablesync.c
* PostgreSQL logical replication
*
- * Copyright (c) 2012-2016, PostgreSQL Global Development Group
+ * Copyright (c) 2012-2017, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/backend/replication/logical/tablesync.c
diff --git a/src/include/catalog/pg_subscription_rel.h b/src/include/catalog/pg_subscription_rel.h
index e0cc66cb..9f4f152 100644
--- a/src/include/catalog/pg_subscription_rel.h
+++ b/src/include/catalog/pg_subscription_rel.h
@@ -4,7 +4,7 @@
* Local info about tables that come from the publisher of a
* subscription (pg_subscription_rel).
*
- * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
* -------------------------------------------------------------------------
On Thu, Mar 23, 2017 at 9:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
Logical replication support for initial data copy
+ case T_SQLCmd:
+ if (MyDatabaseId == InvalidOid)
+ ereport(ERROR,
+ (errmsg("not connected to database")));This error message doesn't seem to follow the error message style in docs.
Also It seems a bit unclear to me. So what about replacing it with
something like the following?
ERROR: must connect to database to execute command \"%s\"
Regards,
--
Fujii Masao
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/30/17 14:04, Fujii Masao wrote:
On Thu, Mar 23, 2017 at 9:59 PM, Peter Eisentraut <peter_e@gmx.net> wrote:
Logical replication support for initial data copy
+ case T_SQLCmd: + if (MyDatabaseId == InvalidOid) + ereport(ERROR, + (errmsg("not connected to database")));This error message doesn't seem to follow the error message style in docs.
I have committed an improved message.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers