Binary support for pgoutput plugin
Is there a reason why pgoutput sends data in text format? Seems to me that
sending data in binary would provide a considerable performance improvement.
Dave Cramer
On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote:
Is there a reason why pgoutput sends data in text format? Seems to
me that sending data in binary would provide a considerable
performance improvement.
Are you seeing something that suggests that the text output is taking
a lot of time or other resources?
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
Dave Cramer
On Mon, 3 Jun 2019 at 20:54, David Fetter <david@fetter.org> wrote:
On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote:
Is there a reason why pgoutput sends data in text format? Seems to
me that sending data in binary would provide a considerable
performance improvement.Are you seeing something that suggests that the text output is taking
a lot of time or other resources?Actually it's on the other end that there is improvement. Parsing text
takes much longer for almost everything except ironically text.
To be more transparent there is some desire to use pgoutput for something
other than logical replication. Change Data Capture clients such as
Debezium have a requirement for a stable plugin which is shipped with core
as this is always available in cloud providers offerings. There's no reason
that I am aware of that they cannot use pgoutput for this. There's also no
reason that I am aware that binary outputs can't be supported. The protocol
would have to change slightly and I am working on a POC patch.
Thing is they aren't all written in C so using binary does provide a pretty
substantial win on the decoding end.
Dave
Hi,
On 2019-06-04 15:47:04 -0400, Dave Cramer wrote:
On Mon, 3 Jun 2019 at 20:54, David Fetter <david@fetter.org> wrote:
On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote:
Is there a reason why pgoutput sends data in text format? Seems to
me that sending data in binary would provide a considerable
performance improvement.Are you seeing something that suggests that the text output is taking
a lot of time or other resources?Actually it's on the other end that there is improvement. Parsing text
takes much longer for almost everything except ironically text.
It's on both sides, I'd say. E.g. float (until v12), timestamp, bytea
are all much more expensive to convert from binary to text.
To be more transparent there is some desire to use pgoutput for something
other than logical replication. Change Data Capture clients such as
Debezium have a requirement for a stable plugin which is shipped with core
as this is always available in cloud providers offerings. There's no reason
that I am aware of that they cannot use pgoutput for this.
Except that that's not pgoutput's purpose, and we shouldn't make it
meaningfully more complicated or slower to achieve this. Don't think
there's a conflict in this case though.
There's also no reason that I am aware that binary outputs can't be
supported.
Well, it *does* increase version dependencies, and does make replication
more complicated, because type oids etc cannot be relied to be the same
on source and target side.
The protocol would have to change slightly and I am working
on a POC patch.
Hm, what would have to be changed protocol wise? IIRC that'd just be a
different datum type? Or is that what you mean?
pq_sendbyte(out, 't'); /* 'text' data follows */
IIRC there was code for the binary protocol in a predecessor of
pgoutput.
I think if we were to add binary output - and I think we should - we
ought to only accept a patch if it's also used in core.
Greetings,
Andres Freund
Dave Cramer
On Tue, 4 Jun 2019 at 16:30, Andres Freund <andres.freund@enterprisedb.com>
wrote:
Hi,
On 2019-06-04 15:47:04 -0400, Dave Cramer wrote:
On Mon, 3 Jun 2019 at 20:54, David Fetter <david@fetter.org> wrote:
On Mon, Jun 03, 2019 at 10:49:54AM -0400, Dave Cramer wrote:
Is there a reason why pgoutput sends data in text format? Seems to
me that sending data in binary would provide a considerable
performance improvement.Are you seeing something that suggests that the text output is taking
a lot of time or other resources?Actually it's on the other end that there is improvement. Parsing text
takes much longer for almost everything except ironically text.
It's on both sides, I'd say. E.g. float (until v12), timestamp, bytea
are all much more expensive to convert from binary to text.To be more transparent there is some desire to use pgoutput for something
other than logical replication. Change Data Capture clients such as
Debezium have a requirement for a stable plugin which is shipped withcore
as this is always available in cloud providers offerings. There's no
reason
that I am aware of that they cannot use pgoutput for this.
Except that that's not pgoutput's purpose, and we shouldn't make it
meaningfully more complicated or slower to achieve this. Don't think
there's a conflict in this case though.
agreed, my intent was to slightly bend it to my will :)
There's also no reason that I am aware that binary outputs can't be
supported.Well, it *does* increase version dependencies, and does make replication
more complicated, because type oids etc cannot be relied to be the same
on source and target side.I was about to agree with this but if the type oids change from source to
target you
still can't decode the text version properly. Unless I mis-understand
something here ?
The protocol would have to change slightly and I am working
on a POC patch.Hm, what would have to be changed protocol wise? IIRC that'd just be a
different datum type? Or is that what you mean?
pq_sendbyte(out, 't'); /* 'text' data follows */I haven't really thought this through completely but one place JDBC has
problems with binary is with
timestamps with timezone as we don't know which timezone to use. Is it safe
to assume everything is in UTC
since the server stores in UTC ? Then there are UDF's. My original thought
was to use options to send in the
types that I wanted in binary, everything else could be sent as text.
IIRC there was code for the binary protocol in a predecessor of
pgoutput.
Hmmm that might be good place to start. I will do some digging through git
history
I think if we were to add binary output - and I think we should - we
ought to only accept a patch if it's also used in core.
Certainly; as not doing so would make my work completely irrelevant for my
purpose.
Thanks,
Dave
Import Notes
Reply to msg id not found: 20190604203009.bbwmiepag7jp5ahj@alap3.anarazel.de
Hi,
On 2019-06-04 16:39:32 -0400, Dave Cramer wrote:
On Tue, 4 Jun 2019 at 16:30, Andres Freund <andres.freund@enterprisedb.com>
wrote:There's also no reason that I am aware that binary outputs can't be
supported.Well, it *does* increase version dependencies, and does make replication
more complicated, because type oids etc cannot be relied to be the same
on source and target side.I was about to agree with this but if the type oids change from source
to target you still can't decode the text version properly. Unless I
mis-understand something here ?
The text format doesn't care about oids. I don't see how it'd be a
problem? Note that some people *intentionally* use different types from
source to target system when logically replicating. So you can't rely on
the target table's types under any circumstance.
I think you really have to use the textual type which we already write
out (cf logicalrep_write_typ()) to call the binary input functions. And
you can send only data as binary that's from builtin types - otherwise
there's no guarantee at all that the target system has something
compatible. And even if you just assumed that all extensions etc are
present, you can't transport arrays / composite types in binary: For
hard to discern reasons we a) embed type oids in them b) verify them. b)
won't ever work for non-builtin types, because oids are assigned
dynamically.
I think if we were to add binary output - and I think we should - we
ought to only accept a patch if it's also used in core.Certainly; as not doing so would make my work completely irrelevant for my
purpose.
What I mean is that the builtin logical replication would have to use
this on the receiving side too.
Greetings,
Andres Freund
On Tue, 4 Jun 2019 at 16:46, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-04 16:39:32 -0400, Dave Cramer wrote:
On Tue, 4 Jun 2019 at 16:30, Andres Freund <
andres.freund@enterprisedb.com>
wrote:
There's also no reason that I am aware that binary outputs can't be
supported.Well, it *does* increase version dependencies, and does make
replication
more complicated, because type oids etc cannot be relied to be the same
on source and target side.I was about to agree with this but if the type oids change from source
to target you still can't decode the text version properly. Unless I
mis-understand something here ?The text format doesn't care about oids. I don't see how it'd be a
problem? Note that some people *intentionally* use different types from
source to target system when logically replicating. So you can't rely on
the target table's types under any circumstance.I think you really have to use the textual type which we already write
out (cf logicalrep_write_typ()) to call the binary input functions. And
you can send only data as binary that's from builtin types - otherwise
there's no guarantee at all that the target system has something
compatible. And even if you just assumed that all extensions etc are
present, you can't transport arrays / composite types in binary: For
hard to discern reasons we a) embed type oids in them b) verify them. b)
won't ever work for non-builtin types, because oids are assigned
dynamically.
I figured arrays and UDT's would be problematic.
I think if we were to add binary output - and I think we should - we
ought to only accept a patch if it's also used in core.Certainly; as not doing so would make my work completely irrelevant for
my
purpose.
What I mean is that the builtin logical replication would have to use
this on the receiving side too.Got it, thanks for validating that the idea isn't nuts. Now I *have* to
produce a POC.
Thanks,
Dave
Show quoted text
On 6/4/19 4:39 PM, Dave Cramer wrote:
I haven't really thought this through completely but one place JDBC has
problems with binary is with
timestamps with timezone as we don't know which timezone to use. Is it safe
to assume everything is in UTC
since the server stores in UTC ?
PL/Java, when converting to the Java 8 java.time types (because those
are sane), will turn a timestamp with timezone into an OffsetDateTime
with explicit offset zero (UTC), no matter what timezone may have been
used when the value was input (as you've observed, there's no way to
recover that). In the return direction, if given an OffsetDateTime
with any nonzero offset, it will adjust the value to UTC for postgres.
So, yes, say I.
Regards,
-Chap
On Tue, Jun 04, 2019 at 04:55:33PM -0400, Dave Cramer wrote:
On Tue, 4 Jun 2019 at 16:46, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-04 16:39:32 -0400, Dave Cramer wrote:
On Tue, 4 Jun 2019 at 16:30, Andres Freund <
andres.freund@enterprisedb.com>
wrote:
There's also no reason that I am aware that binary outputs can't be
supported.Well, it *does* increase version dependencies, and does make
replication
more complicated, because type oids etc cannot be relied to be the same
on source and target side.I was about to agree with this but if the type oids change from source
to target you still can't decode the text version properly. Unless I
mis-understand something here ?The text format doesn't care about oids. I don't see how it'd be a
problem? Note that some people *intentionally* use different types from
source to target system when logically replicating. So you can't rely on
the target table's types under any circumstance.I think you really have to use the textual type which we already write
out (cf logicalrep_write_typ()) to call the binary input functions. And
you can send only data as binary that's from builtin types - otherwise
there's no guarantee at all that the target system has something
compatible. And even if you just assumed that all extensions etc are
present, you can't transport arrays / composite types in binary: For
hard to discern reasons we a) embed type oids in them b) verify them. b)
won't ever work for non-builtin types, because oids are assigned
dynamically.I figured arrays and UDT's would be problematic.
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.
Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778
Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.
I think you recall wrongly. It's obviously possible that we have bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have to do
endianess conversions, but that doesn't make it non-standardized.
- Andres
On Tue, 4 Jun 2019 at 18:08, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.I think you recall wrongly. It's obviously possible that we have bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have to do
endianess conversions, but that doesn't make it non-standardized.
Additionally there are a number of drivers that already know how to handle
our binary types.
I don't really think there's a win here. I also want to keep the changes
small .
Dave
Hi,
On 05/06/2019 00:08, Andres Freund wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.I think you recall wrongly. It's obviously possible that we have bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have to do
endianess conversions, but that doesn't make it non-standardized.
Yeah, there are really 3 formats of data we have, text protocol, binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.
FWIW I don't think the code for binary format was included in original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for adding that.
That said, pglogical has code which handles this (I guess Andres means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer in
https://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions
https://www.2ndQuadrant.com/
Hi,
On Wed, 5 Jun 2019 at 07:18, Petr Jelinek <petr.jelinek@2ndquadrant.com>
wrote:
Hi,
On 05/06/2019 00:08, Andres Freund wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.I think you recall wrongly. It's obviously possible that we have bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have to do
endianess conversions, but that doesn't make it non-standardized.Yeah, there are really 3 formats of data we have, text protocol, binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.FWIW I don't think the code for binary format was included in original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for adding that.
Yes, I looked through the public history and could not find it. Thanks for
confirming.
That said, pglogical has code which handles this (I guess Andres means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer inhttps://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.
Thanks for the tip!
Dave Cramer
Show quoted text
On Wed, 5 Jun 2019 at 07:21, Dave Cramer <davecramer@gmail.com> wrote:
Hi,
On Wed, 5 Jun 2019 at 07:18, Petr Jelinek <petr.jelinek@2ndquadrant.com>
wrote:Hi,
On 05/06/2019 00:08, Andres Freund wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary format is
not standardized across, for example, big- and little-endian machines.I think you recall wrongly. It's obviously possible that we have bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have to do
endianess conversions, but that doesn't make it non-standardized.Yeah, there are really 3 formats of data we have, text protocol, binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.FWIW I don't think the code for binary format was included in original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for adding that.Yes, I looked through the public history and could not find it. Thanks for
confirming.That said, pglogical has code which handles this (I guess Andres means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer inhttps://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.Thanks for the tip!
this seems completely ignored. What was the intent?
Dave
Hi
On June 5, 2019 8:51:10 AM PDT, Dave Cramer <davecramer@gmail.com> wrote:
On Wed, 5 Jun 2019 at 07:21, Dave Cramer <davecramer@gmail.com> wrote:
Hi,
On Wed, 5 Jun 2019 at 07:18, Petr Jelinek
<petr.jelinek@2ndquadrant.com>
wrote:
Hi,
On 05/06/2019 00:08, Andres Freund wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary formatis
not standardized across, for example, big- and little-endian
machines.
I think you recall wrongly. It's obviously possible that we have
bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have todo
endianess conversions, but that doesn't make it non-standardized.
Yeah, there are really 3 formats of data we have, text protocol,
binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.FWIW I don't think the code for binary format was included in
original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for addingthat.
Yes, I looked through the public history and could not find it.
Thanks for
confirming.
That said, pglogical has code which handles this (I guess Andres
means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer inhttps://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.
Thanks for the tip!
this seems completely ignored. What was the intent?
That's about the output of the plugin, not the datatypes. And independent of text/binary output, the protocol contains non-printable chars.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Hi,
On Wed, 5 Jun 2019 at 12:01, Andres Freund <andres@anarazel.de> wrote:
Hi
On June 5, 2019 8:51:10 AM PDT, Dave Cramer <davecramer@gmail.com> wrote:
On Wed, 5 Jun 2019 at 07:21, Dave Cramer <davecramer@gmail.com> wrote:
Hi,
On Wed, 5 Jun 2019 at 07:18, Petr Jelinek
<petr.jelinek@2ndquadrant.com>
wrote:
Hi,
On 05/06/2019 00:08, Andres Freund wrote:
Hi,
On 2019-06-05 00:05:02 +0200, David Fetter wrote:
Would it make sense to work toward a binary format that's not
architecture-specific? I recall from COPY that our binary formatis
not standardized across, for example, big- and little-endian
machines.
I think you recall wrongly. It's obviously possible that we have
bugs
around this, but output/input routines are supposed to handle a
endianess independent format. That usually means that you have todo
endianess conversions, but that doesn't make it non-standardized.
Yeah, there are really 3 formats of data we have, text protocol,
binary
network protocol and internal on disk format. The internal on disk
format will not work across big/little-endian but network binary
protocol will.FWIW I don't think the code for binary format was included in
original
logical replication patch (I really tried to keep it as minimal as
possible), but the code and protocol is pretty much ready for addingthat.
Yes, I looked through the public history and could not find it.
Thanks for
confirming.
That said, pglogical has code which handles this (I guess Andres
means
that by predecessor of pgoutput) so if you look for example at the
write_tuple/read_tuple/decide_datum_transfer inhttps://github.com/2ndQuadrant/pglogical/blob/REL2_x_STABLE/pglogical_proto_native.c
that can help you give some ideas on how to approach this.
Thanks for the tip!
Looking at:
this seems completely ignored. What was the intent?
That's about the output of the plugin, not the datatypes. And independent
of text/binary output, the protocol contains non-printable chars.Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional in Java.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if we were
going to bump the protocol?
Dave
Hi,
On 2019-06-05 18:47:57 -0400, Dave Cramer wrote:
So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional in Java.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if we were
going to bump the protocol?
I'm pretty strongly opposed to this. What's the limiting factor when
adding such information? I think clients that want something like this
ought to query the database for catalog information when getting schema
information.
Greetings,
Andres Freund
Hi,
On Wed, 5 Jun 2019 at 18:50, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-05 18:47:57 -0400, Dave Cramer wrote:
So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional inJava.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if wewere
going to bump the protocol?
I'm pretty strongly opposed to this. What's the limiting factor when
adding such information? I think clients that want something like this
ought to query the database for catalog information when getting schema
information.
I'm not intimately familiar with their code. I will query them more about
the ask.
I am curious why you are "strongly" opposed however. We already have the
information. Adding doesn't seem onerous.
Dave
Hi,
On 2019-06-05 19:05:05 -0400, Dave Cramer wrote:
I am curious why you are "strongly" opposed however. We already have the
information. Adding doesn't seem onerous.
(thought I'd already replied with this)
The problem is that I don't recognize a limiting principle:
If we want NOT NULL information for clients, why don't we include the
underlying types for arrays, and the fields in composite types? What
about foreign keys? And unique keys?
And then we suddenly need tracking for all these, so we don't always
send out that information when we previously already did - and in some
of the cases there's no infrastructure for that.
I just don't quite buy that the output plugin build for pg's logical
replication needs is a good place to include a continually increasing
amount of metadata that logical replication doesn't need. That's going
to add overhead and make the code more complicated.
Greetings,
Andres Freund
On 06/07/19 19:27, Andres Freund wrote:
The problem is that I don't recognize a limiting principle:
If we want NOT NULL information for clients, why don't we include the
underlying types for arrays, and the fields in composite types? What
about foreign keys? And unique keys?
This reminds me of an idea I had for a future fe/be protocol version,
right after a talk by Alyssa Ritchie and Henrietta Dombrovskaya at the
last 2Q PGConf. [1]https://www.2qpgconf.com/schedule/information-exchange-techniques-for-javapostgresql-applications/
It seems they had ended up designing a whole 'nother "protocol level"
involving queries wrapping their results as JSON and an app layer that
unwraps again, after trying a simpler first approach that was foiled by the
inability to see into arrays and anonymous record types in the 'describe'
response.
I thought, in a new protocol rev, why not let the driver send additional
'describe' messages after the first one, to drill into structure of
individual columns mentioned in the first response, before sending the
'execute' message?
If it doesn't want the further detail, it doesn't have to ask.
And then we suddenly need tracking for all these, so we don't always
send out that information when we previously already did
If it's up to the client driver, it can track what it needs or already has.
I haven't looked too deeply into the replication protocol ... it happens
under a kind of copy-both, right?, so maybe there's a way for the receiver
to send some inquiries back, but maybe in a windowed, full-duplex way where
it might have to buffer some incoming messages before getting the response
to an inquiry message it sent.
Would those be thinkable thoughts for a future protocol rev?
Regards,
-Chap
[1]: https://www.2qpgconf.com/schedule/information-exchange-techniques-for-javapostgresql-applications/
https://www.2qpgconf.com/schedule/information-exchange-techniques-for-javapostgresql-applications/
Hi,
On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
It seems they had ended up designing a whole 'nother "protocol level"
involving queries wrapping their results as JSON and an app layer that
unwraps again, after trying a simpler first approach that was foiled by the
inability to see into arrays and anonymous record types in the 'describe'
response.
I suspect quite a few people would have to have left the projectbefore
this would happen.
I thought, in a new protocol rev, why not let the driver send additional
'describe' messages after the first one, to drill into structure of
individual columns mentioned in the first response, before sending the
'execute' message?If it doesn't want the further detail, it doesn't have to ask.
And then we suddenly need tracking for all these, so we don't always
send out that information when we previously already didIf it's up to the client driver, it can track what it needs or already has.
I haven't looked too deeply into the replication protocol ... it happens
under a kind of copy-both, right?, so maybe there's a way for the receiver
to send some inquiries back, but maybe in a windowed, full-duplex way where
it might have to buffer some incoming messages before getting the response
to an inquiry message it sent.
That'd be a *lot* of additional complexity, and pretty much prohibitive
from a performance POV. We'd have to not continue decoding on the server
side *all* the time to give the client a chance to inquire additional
information.
Greetings,
Andres Freund
On 06/07/19 21:01, Andres Freund wrote:
On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
It seems they had ended up designing a whole 'nother "protocol level"
involving queries wrapping their results as JSON and an app layer that
unwraps again, after trying a simpler first approach that was foiled by the
inability to see into arrays and anonymous record types in the 'describe'
response.I suspect quite a few people would have to have left the projectbefore
this would happen.
I'm not sure I understand what you're getting at. The "whole 'nother
protocol" was something they actually implemented, at the application
level, by rewriting their queries to produce JSON and their client to
unwrap it. It wasn't proposed to go into postgres ... but it was a
workaround they were forced into by the current protocol's inability
to tell them what they were getting.
That'd be a *lot* of additional complexity, and pretty much prohibitive
from a performance POV. We'd have to not continue decoding on the server
side *all* the time to give the client a chance to inquire additional
information.
Does anything travel in the client->server direction during replication?
I thought I saw CopyBoth mentioned. Is there a select()/poll() being done
so those messages can be received?
It does seem that the replication protocol would be the tougher problem.
For the regular extended-query protocol, it seems like allowing an extra
Describe or two before Execute might not be awfully hard.
Regards,
-Chap
Hi,
On 2019-06-07 21:16:12 -0400, Chapman Flack wrote:
On 06/07/19 21:01, Andres Freund wrote:
On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
That'd be a *lot* of additional complexity, and pretty much prohibitive
from a performance POV. We'd have to not continue decoding on the server
side *all* the time to give the client a chance to inquire additional
information.Does anything travel in the client->server direction during replication?
I thought I saw CopyBoth mentioned. Is there a select()/poll() being done
so those messages can be received?
Yes, acknowledgements of how far data has been received (and how far
processed), which is then used to release resources (WAL, xid horizon)
and allow synchronous replication to block until something has been
received.
- Andres
On Fri, Jun 07, 2019 at 06:01:12PM -0700, Andres Freund wrote:
Hi,
On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
It seems they had ended up designing a whole 'nother "protocol level"
involving queries wrapping their results as JSON and an app layer that
unwraps again, after trying a simpler first approach that was foiled by the
inability to see into arrays and anonymous record types in the 'describe'
response.I suspect quite a few people would have to have left the projectbefore
this would happen.I thought, in a new protocol rev, why not let the driver send additional
'describe' messages after the first one, to drill into structure of
individual columns mentioned in the first response, before sending the
'execute' message?If it doesn't want the further detail, it doesn't have to ask.
And then we suddenly need tracking for all these, so we don't always
send out that information when we previously already didIf it's up to the client driver, it can track what it needs or already has.
I haven't looked too deeply into the replication protocol ... it happens
under a kind of copy-both, right?, so maybe there's a way for the receiver
to send some inquiries back, but maybe in a windowed, full-duplex way where
it might have to buffer some incoming messages before getting the response
to an inquiry message it sent.That'd be a *lot* of additional complexity, and pretty much prohibitive
from a performance POV. We'd have to not continue decoding on the server
side *all* the time to give the client a chance to inquire additional
information.
I kinda agree with this, and I think it's an argument why replication
solutions that need such additional metadata (e.g. because they have no
database to query) should not rely on pgoutput but should invent their own
decoding plugin. Which is why it's a plugin.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
This should have gone to hackers as well
---------- Forwarded message ---------
From: Dave Cramer <davecramer@gmail.com>
Date: Sat, Jun 8, 2019, 6:41 PM
Subject: Re: Binary support for pgoutput plugin
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
On Sat, Jun 8, 2019, 6:27 PM Tomas Vondra, <tomas.vondra@2ndquadrant.com>
wrote:
On Fri, Jun 07, 2019 at 06:01:12PM -0700, Andres Freund wrote:
Hi,
On 2019-06-07 20:52:38 -0400, Chapman Flack wrote:
It seems they had ended up designing a whole 'nother "protocol level"
involving queries wrapping their results as JSON and an app layer that
unwraps again, after trying a simpler first approach that was foiled bythe
inability to see into arrays and anonymous record types in the
'describe'
response.
I suspect quite a few people would have to have left the projectbefore
this would happen.I thought, in a new protocol rev, why not let the driver send additional
'describe' messages after the first one, to drill into structure of
individual columns mentioned in the first response, before sending the
'execute' message?If it doesn't want the further detail, it doesn't have to ask.
And then we suddenly need tracking for all these, so we don't always
send out that information when we previously already didIf it's up to the client driver, it can track what it needs or already
has.
I haven't looked too deeply into the replication protocol ... it happens
under a kind of copy-both, right?, so maybe there's a way for thereceiver
to send some inquiries back, but maybe in a windowed, full-duplex way
where
it might have to buffer some incoming messages before getting the
response
to an inquiry message it sent.
That'd be a *lot* of additional complexity, and pretty much prohibitive
from a performance POV. We'd have to not continue decoding on the server
side *all* the time to give the client a chance to inquire additional
information.I kinda agree with this, and I think it's an argument why replication
solutions that need such additional metadata (e.g. because they have no
database to query) should not rely on pgoutput but should invent their own
decoding plugin. Which is why it's a plugin.
So the reason we are discussing using pgoutput plugin is because it is part
of core and guaranteed to be in cloud providers solutions. I'm trying to
find a balance here of using what we have as opposed to burdening core to
take on additional code to take care of. Not sending the metadata is not a
deal breaker but i can see some value in it.
Dave
Show quoted text
Import Notes
Reply to msg id not found: CADK3HH+XJea1UDecRJUO-nOVLwQdtTgnrd9O+NCuCtGb3sOMA@mail.gmail.com
Hi,
On 2019-06-08 19:41:34 -0400, Dave Cramer wrote:
So the reason we are discussing using pgoutput plugin is because it is part
of core and guaranteed to be in cloud providers solutions.
IMO people needing this should then band together and write one that's
suitable, rather than trying to coerce test_decoding and now pgoutput
into something they're not made for.
Greetings,
Andres Freund
On Sat, 8 Jun 2019 at 20:09, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-08 19:41:34 -0400, Dave Cramer wrote:
So the reason we are discussing using pgoutput plugin is because it is
part
of core and guaranteed to be in cloud providers solutions.
IMO people needing this should then band together and write one that's
suitable, rather than trying to coerce test_decoding and now pgoutput
into something they're not made for.
At the moment it would look a lot like pgoutput. For now I'm fine with no
changes to pgoutput other than binary
Once we have some real use cases we can look at writing a new one. I would
imagine we would actually start with
pgoutput and just add to it.
Thanks,
Dave
On Sat, Jun 08, 2019 at 08:40:43PM -0400, Dave Cramer wrote:
On Sat, 8 Jun 2019 at 20:09, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-08 19:41:34 -0400, Dave Cramer wrote:
So the reason we are discussing using pgoutput plugin is because it is
part
of core and guaranteed to be in cloud providers solutions.
IMO people needing this should then band together and write one that's
suitable, rather than trying to coerce test_decoding and now pgoutput
into something they're not made for.At the moment it would look a lot like pgoutput. For now I'm fine with no
changes to pgoutput other than binary
Once we have some real use cases we can look at writing a new one. I would
imagine we would actually start with
pgoutput and just add to it.
I understand the desire to make this work for managed cloud environments,
we support quite a few customers who would benefit from it. But pgoutput
is meant specifically for built-in replication, and adding complexity that
is useless for that use case does not seem like a good tradeoff.
From this POV the binary mode is fine, because it'd benefit pgoutput, but
the various other stuff mentioned here (e.g. nullability) is not.
And if we implement a new plugin for use by out-of-core stuff, I guess
we'd probably done it in an extension. But even having it in contrib would
not make it automatically installed on managed systems, because AFAIK the
various providers only allow whitelisted extensions. At which point
there's there's little difference compared to external extensions.
I think the best party to implement such extension is whoever implements
such replication system (say Debezium), because they are the ones who know
which format / behavior would work for them. And they can also show the
benefit to their users, who can then push the cloud providers to install
the extension. Of course, that'll take a long time (but it's unclear how
long), and until then they'll have to provide some fallback.
This is a bit of a chicken-egg problem, with three parties - our project,
projects building on logical replication and cloud providers. And no
matter how you slice it, the party implementing it has only limited (if
any) control over what the other parties allow.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
So back to binary output.
From what I can tell the place to specify binary options would be in the
create publication and or in replication slots?
The challenge as I see it is that the subscriber would have to be able to
decode binary output.
Any thoughts on how to handle this? At the moment I'm assuming that this
would only work for subscribers that knew how to handle binary.
Regards,
Dave
Show quoted text
Hi,
On 10/06/2019 13:27, Dave Cramer wrote:
So back to binary output.
From what I can tell the place to specify binary options would be in the
create publication and or in replication slots?The challenge as I see it is that the subscriber would have to be able
to decode binary output.Any thoughts on how to handle this? At the moment I'm assuming that this
would only work for subscribers that knew how to handle binary.
Given that we don't need to write anything extra to WAL on upstream to
support binary output, why not just have the request for binary data as
an option for the pgoutput and have it chosen dynamically? Then it's the
subscriber who asks for binary output via option(s) to START_REPLICATION.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions
https://www.2ndQuadrant.com/
OK, before I go too much further down this rabbit hole I'd like feedback on
the current code. See attached patch
There is one obvious hack where in binary mode I reset the input cursor to
allow the binary input to be re-read
From what I can tell the alternative is to convert the data in
logicalrep_read_tuple but that would require moving a lot of the logic
currently in worker.c to proto.c. This seems minimally invasive.
and thanks Petr for the tip to use pglogical for ideas.
Thanks,
Dave Cramer
Show quoted text
Attachments:
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From 4bb31d06870dce5ed5e6a781ab13e30600069a7e Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 121 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 35 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 198 insertions(+), 87 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 43edfef089..e838b376c6 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -363,14 +384,14 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Modify slot with user data provided as C strings.
+ * Modify slot with user data provided.
* This is somewhat similar to heap_modify_tuple but also calls the type
- * input function on the user data as the input is the text representation
- * of the types.
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -398,21 +419,43 @@ slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -576,8 +619,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -599,7 +646,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -679,9 +726,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -709,8 +759,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -741,7 +791,7 @@ apply_handle_update(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
ExecCopySlot(remoteslot, localslot);
- slot_modify_cstrings(remoteslot, rel, newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -799,8 +849,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -828,7 +881,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d317fd7006..4d40c7cd37 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "utils/inval.h"
#include "utils/int8.h"
+#include "utils/builtins.h"
#include "utils/memutils.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -88,11 +89,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -138,6 +143,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -172,7 +193,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -344,7 +366,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -354,7 +377,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -363,7 +387,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 9c0000cc59..aa98a03864 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
On Mon, 10 Jun 2019 at 07:49, Petr Jelinek <petr.jelinek@2ndquadrant.com>
wrote:
Hi,
On 10/06/2019 13:27, Dave Cramer wrote:
So back to binary output.
From what I can tell the place to specify binary options would be in the
create publication and or in replication slots?The challenge as I see it is that the subscriber would have to be able
to decode binary output.Any thoughts on how to handle this? At the moment I'm assuming that this
would only work for subscribers that knew how to handle binary.Given that we don't need to write anything extra to WAL on upstream to
support binary output, why not just have the request for binary data as
an option for the pgoutput and have it chosen dynamically? Then it's the
subscriber who asks for binary output via option(s) to START_REPLICATION.
If I understand this correctly this would add something to the CREATE/ALTER
SUBSCRIPTION commands in the WITH clause.
Additionally another column would be required for pg_subscription for the
binary option.
Does it make sense to add an options column which would just be a comma
separated string?
Not that I have future options in mind but seems like something that might
come up in the future.
Dave Cramer
Show quoted text
On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote:
On Mon, 10 Jun 2019 at 07:49, Petr Jelinek <petr.jelinek@2ndquadrant.com>
wrote:Hi,
On 10/06/2019 13:27, Dave Cramer wrote:
So back to binary output.
From what I can tell the place to specify binary options would be in the
create publication and or in replication slots?The challenge as I see it is that the subscriber would have to be able
to decode binary output.Any thoughts on how to handle this? At the moment I'm assuming that this
would only work for subscribers that knew how to handle binary.Given that we don't need to write anything extra to WAL on upstream to
support binary output, why not just have the request for binary data as
an option for the pgoutput and have it chosen dynamically? Then it's the
subscriber who asks for binary output via option(s) to START_REPLICATION.If I understand this correctly this would add something to the CREATE/ALTER
SUBSCRIPTION commands in the WITH clause.
Additionally another column would be required for pg_subscription for the
binary option.
Does it make sense to add an options column which would just be a comma
separated string?
Not that I have future options in mind but seems like something that might
come up in the future.
I'd just add a boolean column to the catalog. That's what I did in the
patch adding support for streaming in-progress transactions. I don't think
we expect many additional parameters, so it makes little sense to optimize
for that case.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Dave Cramer
On Fri, 14 Jun 2019 at 14:36, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote:
On Mon, 10 Jun 2019 at 07:49, Petr Jelinek <petr.jelinek@2ndquadrant.com>
wrote:Hi,
On 10/06/2019 13:27, Dave Cramer wrote:
So back to binary output.
From what I can tell the place to specify binary options would be in
the
create publication and or in replication slots?
The challenge as I see it is that the subscriber would have to be able
to decode binary output.Any thoughts on how to handle this? At the moment I'm assuming that
this
would only work for subscribers that knew how to handle binary.
Given that we don't need to write anything extra to WAL on upstream to
support binary output, why not just have the request for binary data as
an option for the pgoutput and have it chosen dynamically? Then it's the
subscriber who asks for binary output via option(s) toSTART_REPLICATION.
If I understand this correctly this would add something to the
CREATE/ALTER
SUBSCRIPTION commands in the WITH clause.
Additionally another column would be required for pg_subscription for the
binary option.
Does it make sense to add an options column which would just be a comma
separated string?
Not that I have future options in mind but seems like something that might
come up in the future.I'd just add a boolean column to the catalog. That's what I did in the
patch adding support for streaming in-progress transactions. I don't think
we expect many additional parameters, so it makes little sense to optimize
for that case.
Which is what I have done. Thanks
I've attached both patches for comments.
I still have to add documentation.
Regards,
Dave
Attachments:
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From 7783e793b45695c1a9f04add84b132aa80af9ac1 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/2] add binary column to pg_subscription bump catversion
support create and alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 3 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 44 insertions(+), 10 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 78a103cdb9..4b46198e2b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1018,7 +1018,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index f13dce90a1..2aac57f698 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +97,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -182,6 +188,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -331,8 +342,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -341,7 +354,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -398,6 +411,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -673,10 +687,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -701,6 +718,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -712,7 +736,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -750,7 +775,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -787,7 +812,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..9aec824a18 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,6 +400,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -421,6 +422,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index e838b376c6..b75fbd6bd6 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1711,7 +1711,7 @@ ApplyWorkerMain(Datum main_arg)
{
char *syncslotname;
- /* This is table synchroniation worker, call initial sync. */
+ /* This is table synchronization worker, call initial sync. */
syncslotname = LogicalRepSyncTableStart(&origin_startpos);
/* The slot name needs to be allocated in permanent memory context. */
@@ -1778,6 +1778,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 4b8ee1cbdb..4b637bcbae 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 201906041
+#define CATALOG_VERSION_NO 201906120
#endif
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..bb44e5e45c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 86a8130051..55a5ff5fa1 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From 4bb31d06870dce5ed5e6a781ab13e30600069a7e Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/2] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 121 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 35 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 198 insertions(+), 87 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 43edfef089..e838b376c6 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -363,14 +384,14 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Modify slot with user data provided as C strings.
+ * Modify slot with user data provided.
* This is somewhat similar to heap_modify_tuple but also calls the type
- * input function on the user data as the input is the text representation
- * of the types.
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -398,21 +419,43 @@ slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -576,8 +619,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -599,7 +646,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -679,9 +726,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -709,8 +759,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -741,7 +791,7 @@ apply_handle_update(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
ExecCopySlot(remoteslot, localslot);
- slot_modify_cstrings(remoteslot, rel, newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -799,8 +849,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -828,7 +881,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d317fd7006..4d40c7cd37 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "utils/inval.h"
#include "utils/int8.h"
+#include "utils/builtins.h"
#include "utils/memutils.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -88,11 +89,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -138,6 +143,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -172,7 +193,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -344,7 +366,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -354,7 +377,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -363,7 +387,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 9c0000cc59..aa98a03864 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
On Fri, 14 Jun 2019 at 15:42, Dave Cramer <davecramer@gmail.com> wrote:
Dave Cramer
On Fri, 14 Jun 2019 at 14:36, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:On Wed, Jun 12, 2019 at 10:35:48AM -0400, Dave Cramer wrote:
On Mon, 10 Jun 2019 at 07:49, Petr Jelinek <petr.jelinek@2ndquadrant.com
wrote:
Hi,
On 10/06/2019 13:27, Dave Cramer wrote:
So back to binary output.
From what I can tell the place to specify binary options would be in
the
create publication and or in replication slots?
The challenge as I see it is that the subscriber would have to be
able
to decode binary output.
Any thoughts on how to handle this? At the moment I'm assuming that
this
would only work for subscribers that knew how to handle binary.
Given that we don't need to write anything extra to WAL on upstream to
support binary output, why not just have the request for binary data as
an option for the pgoutput and have it chosen dynamically? Then it'sthe
subscriber who asks for binary output via option(s) to
START_REPLICATION.
If I understand this correctly this would add something to the
CREATE/ALTER
SUBSCRIPTION commands in the WITH clause.
Additionally another column would be required for pg_subscription for the
binary option.
Does it make sense to add an options column which would just be a comma
separated string?
Not that I have future options in mind but seems like something thatmight
come up in the future.
I'd just add a boolean column to the catalog. That's what I did in the
patch adding support for streaming in-progress transactions. I don't think
we expect many additional parameters, so it makes little sense to optimize
for that case.Which is what I have done. Thanks
I've attached both patches for comments.
I still have to add documentation.Regards,
Dave
Additional patch for documentation.
Dave Cramer
Attachments:
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 2e37c3ce977f5ffec773943921839b3b1cee5dd4 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/3] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 36193d1491..0a43cef488 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6696,6 +6696,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..cc2ff0363f 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
On Wed, 5 Jun 2019 at 18:50, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2019-06-05 18:47:57 -0400, Dave Cramer wrote:
So one of the things they would like added is to get not null information
in the schema record. This is so they can mark the field Optional inJava.
I presume this would also have some uses in other languages. As I
understand it this would require a protocol bump. If this were to be
accepted are there any outstanding asks that would useful to add if wewere
going to bump the protocol?
I'm pretty strongly opposed to this. What's the limiting factor when
adding such information? I think clients that want something like this
ought to query the database for catalog information when getting schema
information.
So talking some more to the guys that want to use this for Change Data
Capture they pointed out that if the schema changes while they are offline
there is no way to query the catalog information
Dave
Show quoted text
On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote:
Which is what I have done. Thanks
I've attached both patches for comments.
I still have to add documentation.Additional patch for documentation.
Thank you for the patch! Unfortunately 0002 has some conflicts, could
you please send a rebased version? In the meantime I have few questions:
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
....
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
Maybe I'm missing something, what is the reason of moving pq_getmsgint
out of logicalrep_read_*? Just from the patch it seems that the code is
equivalent.
There is one obvious hack where in binary mode I reset the input
cursor to allow the binary input to be re-read From what I can tell
the alternative is to convert the data in logicalrep_read_tuple but
that would require moving a lot of the logic currently in worker.c to
proto.c. This seems minimally invasive.
Which logic has to be moved for example? Actually it sounds more natural
to me, if this functionality would be in e.g. logicalrep_read_tuple
rather than slot_store_data, since it has something to do with reading
data. And it seems that in pglogical it's also located in
pglogical_read_tuple.
On Sun, 27 Oct 2019 at 11:00, Dmitry Dolgov <9erthalion6@gmail.com> wrote:
On Mon, Jun 17, 2019 at 10:29:26AM -0400, Dave Cramer wrote:
Which is what I have done. Thanks
I've attached both patches for comments.
I still have to add documentation.Additional patch for documentation.
Thank you for the patch! Unfortunately 0002 has some conflicts, could
you please send a rebased version? In the meantime I have few questions:-LogicalRepRelId +void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup) { char action; - LogicalRepRelId relid; - - /* read the relation id */ - relid = pq_getmsgint(in, 4);action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in,
LogicalRepTupleData *newtup)logicalrep_read_tuple(in, newtup);
- return relid;
}....
- relid = logicalrep_read_insert(s, &newtup); + /* read the relation id */ + relid = pq_getmsgint(s, 4); rel = logicalrep_rel_open(relid, RowExclusiveLock); + + logicalrep_read_insert(s, &newtup);Maybe I'm missing something, what is the reason of moving pq_getmsgint
out of logicalrep_read_*? Just from the patch it seems that the code is
equivalent.There is one obvious hack where in binary mode I reset the input
cursor to allow the binary input to be re-read From what I can tell
the alternative is to convert the data in logicalrep_read_tuple but
that would require moving a lot of the logic currently in worker.c to
proto.c. This seems minimally invasive.Which logic has to be moved for example? Actually it sounds more natural
to me, if this functionality would be in e.g. logicalrep_read_tuple
rather than slot_store_data, since it has something to do with reading
data. And it seems that in pglogical it's also located in
pglogical_read_tuple.
Ok, I've rebased and reverted logicalrep_read_insert
As to the last comment, honestly it's been so long I can't remember why I
put that comment in there.
Thanks for reviewing
Dave
Attachments:
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From 457348ad6ff17c1e9051f1ae3a3ec85c803185f1 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/4] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 121 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 35 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 198 insertions(+), 87 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303638..cc505f8c06 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -363,14 +384,14 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Modify slot with user data provided as C strings.
+ * Modify slot with user data provided.
* This is somewhat similar to heap_modify_tuple but also calls the type
- * input function on the user data as the input is the text representation
- * of the types.
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -398,21 +419,43 @@ slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -576,8 +619,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -599,7 +646,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -679,9 +726,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -709,8 +759,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -741,7 +791,7 @@ apply_handle_update(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
ExecCopySlot(remoteslot, localslot);
- slot_modify_cstrings(remoteslot, rel, newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -799,8 +849,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -828,7 +881,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757fca..ee20a6535c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -23,6 +23,7 @@
#include "utils/inval.h"
#include "utils/int8.h"
+#include "utils/builtins.h"
#include "utils/memutils.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -90,11 +91,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -140,6 +145,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +195,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -346,7 +368,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -356,7 +379,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -365,7 +389,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..4d1b7e8c4f 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 6386c3cd05c6896195c2e44e86d43a62ed288e77 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/4] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..a19bb11674 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6763,6 +6763,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..cc2ff0363f 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From 5716d5bc1e743efa33800c1d1ad50a5c4bbf8088 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/4] add binary column to pg_subscription bump catversion
support create and alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a4794a..a744f06f1d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1058,7 +1058,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e67a5889e..07f28e74dd 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +97,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -182,6 +188,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -331,8 +342,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -341,7 +354,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -407,6 +420,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -682,10 +696,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -710,6 +727,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -721,7 +745,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -759,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -796,7 +821,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..9aec824a18 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,6 +400,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -421,6 +422,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index cc505f8c06..6d5c0c1ce1 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 1f6de76e9c..f0b17091e5 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 201910251
+#define CATALOG_VERSION_NO 201910301
#endif
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..bb44e5e45c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934966..641dd1f416 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From 9ecc9fe672ca0ab91dcf8bf76cbc74ca793d3232 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Wed, 30 Oct 2019 09:58:59 -0400
Subject: [PATCH 4/4] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 8859aa8e79..9164d36edc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -161,18 +161,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6d5c0c1ce1..8ad61d9ffb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -620,10 +620,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d1b7e8c4f..b299a2c226 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -91,7 +91,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
On Thu, Oct 31, 2019 at 3:03 AM Dave Cramer <davecramer@gmail.com> wrote:
Ok, I've rebased and reverted logicalrep_read_insert
Hi Dave,
From the code style police (actually just from cfbot, which is set up
to complain about declarations after statements, a bit of C99 we
aren't ready for):
proto.c:557:6: error: ISO C90 forbids mixed declarations and code
[-Werror=declaration-after-statement]
int len = pq_getmsgint(in, 4); /* read length */
^
proto.c:573:6: error: ISO C90 forbids mixed declarations and code
[-Werror=declaration-after-statement]
int len = pq_getmsgint(in, 4); /* read length */
^
On Sun, 3 Nov 2019 at 21:47, Thomas Munro <thomas.munro@gmail.com> wrote:
On Thu, Oct 31, 2019 at 3:03 AM Dave Cramer <davecramer@gmail.com> wrote:
Ok, I've rebased and reverted logicalrep_read_insert
Hi Dave,
From the code style police (actually just from cfbot, which is set up
to complain about declarations after statements, a bit of C99 we
aren't ready for):proto.c:557:6: error: ISO C90 forbids mixed declarations and code
[-Werror=declaration-after-statement]
int len = pq_getmsgint(in, 4); /* read length */
^
proto.c:573:6: error: ISO C90 forbids mixed declarations and code
[-Werror=declaration-after-statement]
int len = pq_getmsgint(in, 4); /* read length */
^
Thomas,
Thanks for the review.
See attached
Attachments:
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From 720c90ea66429d2c22fc422db5b538fd79a4e71d Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 5 Nov 2019 07:14:30 -0500
Subject: [PATCH 5/5] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9164d36edc..5081905a8d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -551,10 +551,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -569,8 +570,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote:
See attached
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
I'm a bit confused, shouldn't be there also
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
in the GetSubscription?
On Fri, 8 Nov 2019 at 11:20, Dmitry Dolgov <9erthalion6@gmail.com> wrote:
On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote:
See attached
--- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg) options.slotname = myslotname; options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; + options.proto.logical.binary = MySubscription->binary;I'm a bit confused, shouldn't be there also
--- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->name = pstrdup(NameStr(subform->subname)); sub->owner = subform->subowner; sub->enabled = subform->subenabled; + sub->binary = subform->subbinary;in the GetSubscription?
yes, I have added this. I will supply an updated patch later.
Now a bigger question(s).
Previously someone mentioned that we need to confirm whether the two
servers are compatible for binary or not.
Checking to make sure the two servers have the same endianness is obvious.
Sizeof int, long, float, double, timestamp (float/int) at a minimum.
this could be done in libpqrcv_startstreaming. The question I have
remaining is do we fall back to text mode if needed or simply fail ?
Dave
On 2019-Nov-11, Dave Cramer wrote:
Previously someone mentioned that we need to confirm whether the two
servers are compatible for binary or not.Checking to make sure the two servers have the same endianness is obvious.
Sizeof int, long, float, double, timestamp (float/int) at a minimum.this could be done in libpqrcv_startstreaming. The question I have
remaining is do we fall back to text mode if needed or simply fail ?
I think it makes more sense to have it fail. If the user wants to retry
in text mode, they can do that easily enough; but if we make it
fall-back automatically and they set up the received wrongly by mistake,
they would pay the performance penalty without noticing.
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, 11 Nov 2019 at 12:04, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
On 2019-Nov-11, Dave Cramer wrote:
Previously someone mentioned that we need to confirm whether the two
servers are compatible for binary or not.Checking to make sure the two servers have the same endianness is
obvious.
Sizeof int, long, float, double, timestamp (float/int) at a minimum.
this could be done in libpqrcv_startstreaming. The question I have
remaining is do we fall back to text mode if needed or simply fail ?I think it makes more sense to have it fail. If the user wants to retry
in text mode, they can do that easily enough; but if we make it
fall-back automatically and they set up the received wrongly by mistake,
they would pay the performance penalty without noticing.
Alvaro,
thanks, after sending this I pretty much came to the same conclusion.
Dave
Show quoted text
On Mon, 11 Nov 2019 at 12:07, Dave Cramer <davecramer@gmail.com> wrote:
On Mon, 11 Nov 2019 at 12:04, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:On 2019-Nov-11, Dave Cramer wrote:
Previously someone mentioned that we need to confirm whether the two
servers are compatible for binary or not.Checking to make sure the two servers have the same endianness is
obvious.
Sizeof int, long, float, double, timestamp (float/int) at a minimum.
this could be done in libpqrcv_startstreaming. The question I have
remaining is do we fall back to text mode if needed or simply fail ?I think it makes more sense to have it fail. If the user wants to retry
in text mode, they can do that easily enough; but if we make it
fall-back automatically and they set up the received wrongly by mistake,
they would pay the performance penalty without noticing.Alvaro,
thanks, after sending this I pretty much came to the same conclusion.
Dave
Following 2 patches address Dmitry's concern and check for compatibility.
Attachments:
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From 74d625f4b533d2ced28268832d7c08799b1eac80 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:38:08 -0500
Subject: [PATCH 7/7] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9aec824a18..19e8070c44 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,7 +400,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -422,9 +421,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ee20a6535c..2c9a5b3eb0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -97,10 +97,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -160,10 +175,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From d2f60d9f6fda82f66c902d4e0c284e87b836f2bd Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:37:35 -0500
Subject: [PATCH 6/7] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee2838cc..8d9889ff19 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
On Mon, Nov 11, 2019 at 11:15:45AM -0500, Dave Cramer wrote:
On Fri, 8 Nov 2019 at 11:20, Dmitry Dolgov <9erthalion6@gmail.com> wrote:On Tue, Nov 05, 2019 at 07:16:10AM -0500, Dave Cramer wrote:
See attached
--- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg) options.slotname = myslotname; options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM; options.proto.logical.publication_names = MySubscription->publications; + options.proto.logical.binary = MySubscription->binary;I'm a bit confused, shouldn't be there also
--- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->name = pstrdup(NameStr(subform->subname)); sub->owner = subform->subowner; sub->enabled = subform->subenabled; + sub->binary = subform->subbinary;in the GetSubscription?
yes, I have added this. I will supply an updated patch later.
Now a bigger question(s).
Well, without this change it wasn't working for me at all. Other than
that yes, it was a small question :)
On 2019-Nov-11, Dave Cramer wrote:
Following 2 patches address Dmitry's concern and check for compatibility.
Please resend the whole patchset, so that the patch tester can verify
the series. (Doing it helps humans, too).
--
�lvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Mon, 11 Nov 2019 at 15:17, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
On 2019-Nov-11, Dave Cramer wrote:
Following 2 patches address Dmitry's concern and check for compatibility.
Please resend the whole patchset, so that the patch tester can verify
the series. (Doing it helps humans, too).
Attached,
Thanks,
Dave
Attachments:
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From 9e551a963b84089f9abfe873a03bc9ab38ecb4cb Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Wed, 30 Oct 2019 09:58:59 -0400
Subject: [PATCH 4/7] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 8859aa8e79..9164d36edc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -161,18 +161,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6d5c0c1ce1..8ad61d9ffb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -620,10 +620,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d1b7e8c4f..b299a2c226 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -91,7 +91,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From 74d625f4b533d2ced28268832d7c08799b1eac80 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:38:08 -0500
Subject: [PATCH 7/7] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9aec824a18..19e8070c44 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,7 +400,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -422,9 +421,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ee20a6535c..2c9a5b3eb0 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -97,10 +97,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -160,10 +175,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From f68f004854215aef2e064e3ded882e94c5cd83e2 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 5 Nov 2019 07:14:30 -0500
Subject: [PATCH 5/7] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9164d36edc..5081905a8d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -551,10 +551,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -569,8 +570,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From d2f60d9f6fda82f66c902d4e0c284e87b836f2bd Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:37:35 -0500
Subject: [PATCH 6/7] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee2838cc..8d9889ff19 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From edd68ea91610563030b32cd0ef26793454ccfb36 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/7] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 121 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 35 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 198 insertions(+), 87 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303638..cc505f8c06 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -363,14 +384,14 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Modify slot with user data provided as C strings.
+ * Modify slot with user data provided.
* This is somewhat similar to heap_modify_tuple but also calls the type
- * input function on the user data as the input is the text representation
- * of the types.
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -398,21 +419,43 @@ slot_modify_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -576,8 +619,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -599,7 +646,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -679,9 +726,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -709,8 +759,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -741,7 +791,7 @@ apply_handle_update(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
ExecCopySlot(remoteslot, localslot);
- slot_modify_cstrings(remoteslot, rel, newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -799,8 +849,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -828,7 +881,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757fca..ee20a6535c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -23,6 +23,7 @@
#include "utils/inval.h"
#include "utils/int8.h"
+#include "utils/builtins.h"
#include "utils/memutils.h"
#include "utils/syscache.h"
#include "utils/varlena.h"
@@ -90,11 +91,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -140,6 +145,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +195,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -346,7 +368,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -356,7 +379,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -365,7 +389,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..4d1b7e8c4f 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From 3cbbe29b3126a241ae8781fa6d0e2579caccf04a Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/7] add binary column to pg_subscription bump catversion
support create and alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a4794a..a744f06f1d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1058,7 +1058,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 1419195766..cf645256dc 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +97,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -182,6 +188,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -331,8 +342,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -341,7 +354,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -407,6 +420,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -677,10 +691,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -705,6 +722,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -716,7 +740,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -754,7 +779,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -791,7 +816,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..9aec824a18 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,6 +400,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -421,6 +422,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index cc505f8c06..6d5c0c1ce1 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1779,6 +1779,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 1f6de76e9c..f0b17091e5 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 201910251
+#define CATALOG_VERSION_NO 201910301
#endif
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..bb44e5e45c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934966..641dd1f416 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From aadf8e2d4b492739dcc27215cf7e1b8a4bc5afed Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/7] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..a19bb11674 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6763,6 +6763,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..cc2ff0363f 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
Hi,
On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote:
Attached,
The latest patch set does not apply correctly. Could you send a
rebase please? I am moving the patch to next CF, waiting on author.
--
Michael
Rebased against head
Dave Cramer
On Sat, 30 Nov 2019 at 20:48, Michael Paquier <michael@paquier.xyz> wrote:
Show quoted text
Hi,
On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote:
Attached,
The latest patch set does not apply correctly. Could you send a
rebase please? I am moving the patch to next CF, waiting on author.
--
Michael
Attachments:
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From 424d9cd3a72c9119aefa4e9481bfd63bb8bfc855 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:38:08 -0500
Subject: [PATCH 7/7] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index c4daea33d2..88693afa88 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,7 +400,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -422,9 +421,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2cb8624e9e..d46095998e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -93,10 +93,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -156,10 +171,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From b5c08d34cb1ba1275d7a175cc82d86d2fef759c9 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/7] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 124 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 34 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 201 insertions(+), 86 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ced0d191c2..9a70649134 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -372,11 +393,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -413,21 +440,43 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -592,8 +641,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -615,7 +668,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -695,9 +748,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -725,8 +781,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -756,8 +812,7 @@ apply_handle_update(StringInfo s)
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, rel,
- newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, localslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -815,8 +870,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -844,7 +902,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b877..2cb8624e9e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -87,11 +87,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -137,6 +141,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +191,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -343,7 +364,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -353,7 +375,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -362,7 +385,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..4d1b7e8c4f 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From d28ef2cb92d852046a375601b6e78b631840f36e Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:37:35 -0500
Subject: [PATCH 6/7] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff499..2d974f82d6 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From edc4ca5e121304bf11713455f77f4735e74424fd Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Wed, 30 Oct 2019 09:58:59 -0400
Subject: [PATCH 4/7] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 8859aa8e79..9164d36edc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -161,18 +161,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 10a776325a..93dc8117a5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -642,10 +642,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d1b7e8c4f..b299a2c226 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -91,7 +91,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From 0b3cb5e7e1ac23602254e9e085e7cc7fe67f5464 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 5 Nov 2019 07:14:30 -0500
Subject: [PATCH 5/7] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9164d36edc..5081905a8d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -551,10 +551,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -569,8 +570,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From e2a7d77bf172538f654d5df9af74a4e92db2eb8b Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/7] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..a19bb11674 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6763,6 +6763,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..cc2ff0363f 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From a5241e0024a9037c1b4fc2da602e8d77aab8742f Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/7] add binary column to pg_subscription bump catversion
support create and alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..3096cf2e5f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1063,7 +1063,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edcfc2..fd5057356a 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +89,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -174,6 +180,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -323,8 +334,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -333,7 +346,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -399,6 +412,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fcd05..c4daea33d2 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,6 +400,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -421,6 +422,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9a70649134..10a776325a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1800,6 +1800,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
index 3a50ba0dfe..1a3fcbf075 100644
--- a/src/include/catalog/catversion.h
+++ b/src/include/catalog/catversion.h
@@ -53,6 +53,6 @@
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 201911241
+#define CATALOG_VERSION_NO 201912021
#endif
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..bb44e5e45c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714eaf0c..49ec831bc4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
On Mon, 2 Dec 2019 at 14:35, Dave Cramer <davecramer@gmail.com> wrote:
Rebased against head
Dave Cramer
On Sat, 30 Nov 2019 at 20:48, Michael Paquier <michael@paquier.xyz> wrote:
Hi,
On Mon, Nov 11, 2019 at 03:24:59PM -0500, Dave Cramer wrote:
Attached,
The latest patch set does not apply correctly. Could you send a
rebase please? I am moving the patch to next CF, waiting on author.
--
Michael
Can I get someone to review this please ?
Dave Cramer
Dave Cramer <davecramer@gmail.com> writes:
Rebased against head
The cfbot's failing to apply this [1]http://cfbot.cputube.org/patch_27_2152.log. It looks like the reason is only
that you included a catversion bump in what you submitted. Protocol is to
*not* do that in submitted patches, but rely on the committer to add it at
the last minute --- otherwise you'll waste a lot of time rebasing the
patch, which is what it needs now.
regards, tom lane
On Fri, 28 Feb 2020 at 18:34, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Dave Cramer <davecramer@gmail.com> writes:
Rebased against head
The cfbot's failing to apply this [1]. It looks like the reason is only
that you included a catversion bump in what you submitted. Protocol is to
*not* do that in submitted patches, but rely on the committer to add it at
the last minute --- otherwise you'll waste a lot of time rebasing the
patch, which is what it needs now.regards, tom lane
rebased and removed the catversion bump.
Attachments:
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From d28ef2cb92d852046a375601b6e78b631840f36e Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:37:35 -0500
Subject: [PATCH 6/7] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff499..2d974f82d6 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From e2a7d77bf172538f654d5df9af74a4e92db2eb8b Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/7] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 55694c4368..a19bb11674 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6763,6 +6763,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..cc2ff0363f 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From a5241e0024a9037c1b4fc2da602e8d77aab8742f Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/7] add binary column to pg_subscription bump catversion
support create and alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f01a6..3096cf2e5f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1063,7 +1063,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edcfc2..fd5057356a 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +89,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -174,6 +180,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -323,8 +334,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -333,7 +346,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -399,6 +412,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fcd05..c4daea33d2 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,6 +400,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -421,6 +422,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 9a70649134..10a776325a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1800,6 +1800,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d897e..bb44e5e45c 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714eaf0c..49ec831bc4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From edc4ca5e121304bf11713455f77f4735e74424fd Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Wed, 30 Oct 2019 09:58:59 -0400
Subject: [PATCH 4/7] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 8859aa8e79..9164d36edc 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -161,18 +161,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 10a776325a..93dc8117a5 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -642,10 +642,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4d1b7e8c4f..b299a2c226 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -91,7 +91,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From 0b3cb5e7e1ac23602254e9e085e7cc7fe67f5464 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 5 Nov 2019 07:14:30 -0500
Subject: [PATCH 5/7] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 9164d36edc..5081905a8d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -551,10 +551,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -569,8 +570,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From b5c08d34cb1ba1275d7a175cc82d86d2fef759c9 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/7] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 124 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 34 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 201 insertions(+), 86 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..8859aa8e79 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -151,7 +153,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -159,14 +161,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -175,7 +173,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -183,7 +180,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -200,26 +197,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -245,14 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -268,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -276,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -292,7 +280,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -441,7 +428,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -457,6 +444,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -492,12 +480,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -518,6 +525,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -528,25 +538,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ced0d191c2..9a70649134 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -296,13 +296,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -327,18 +326,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -372,11 +393,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -413,21 +440,43 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -592,8 +641,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -615,7 +668,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -695,9 +748,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -725,8 +781,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/*
@@ -756,8 +812,7 @@ apply_handle_update(StringInfo s)
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, rel,
- newtup.values, newtup.changed);
+ slot_modify_data(remoteslot, localslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -815,8 +870,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -844,7 +902,7 @@ apply_handle_delete(StringInfo s)
/* Find the tuple using the replica identity index. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/*
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b877..2cb8624e9e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -87,11 +87,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -137,6 +141,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +191,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -343,7 +364,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -353,7 +375,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -362,7 +385,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430af01..4d1b7e8c4f 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -86,16 +90,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 8870721bcd..933038f385 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From 424d9cd3a72c9119aefa4e9481bfd63bb8bfc855 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:38:08 -0500
Subject: [PATCH 7/7] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index c4daea33d2..88693afa88 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -400,7 +400,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -422,9 +421,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2cb8624e9e..d46095998e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -93,10 +93,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -156,10 +171,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
Hi Dave,
On 29/02/2020 18:44, Dave Cramer wrote:
rebased and removed the catversion bump.
Looked into this and it generally seems okay, but I do have one gripe here:
+ tuple->values[i].data = palloc(len + 1); + /* and data */ + + pq_copymsgbytes(in, tuple->values[i].data, len); + tuple->values[i].len = len; + tuple->values[i].cursor = 0; + tuple->values[i].maxlen = len; + /* not strictly necessary but the docs say it is required */ + tuple->values[i].data[len] = '\0'; + break; + } + case 't': /* text formatted value */ + { + tuple->changed[i] = true; + int len = pq_getmsgint(in, 4); /* read length *//* and data */ - tuple->values[i] = palloc(len + 1); - pq_copymsgbytes(in, tuple->values[i], len); - tuple->values[i][len] = '\0'; + tuple->values[i].data = palloc(len + 1); + pq_copymsgbytes(in, tuple->values[i].data, len); + tuple->values[i].data[len] = '\0'; + tuple->values[i].len = len;
The cursor should be set to 0 in the text formatted case too if this is
how we chose to encode data.
However I am not quite convinced I like the StringInfoData usage here.
Why not just change the struct to include additional array of lengths
rather than replacing the existing values array with StringInfoData
array, that seems generally both simpler and should have smaller memory
footprint too, no?
We could also merge the binary and changed arrays into single char array
named something like format (as data can be either unchanged, binary or
text) and just reuse same identifiers we have in protocol.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
On Fri, 6 Mar 2020 at 08:54, Petr Jelinek <petr@2ndquadrant.com> wrote:
Hi Dave,
On 29/02/2020 18:44, Dave Cramer wrote:
rebased and removed the catversion bump.
Looked into this and it generally seems okay, but I do have one gripe here:
+ tuple->values[i].data = palloc(len
+ 1);
+ /* and data */ + + pq_copymsgbytes(in,tuple->values[i].data, len);
+ tuple->values[i].len = len; + tuple->values[i].cursor = 0; + tuple->values[i].maxlen = len; + /* not strictly necessary but thedocs say it is required */
+ tuple->values[i].data[len] = '\0'; + break; + } + case 't': /* text formattedvalue */
+ { + tuple->changed[i] = true; + int len = pq_getmsgint(in, 4); /*read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in,tuple->values[i], len);
- tuple->values[i][len] = '\0'; + tuple->values[i].data = palloc(len+ 1);
+ pq_copymsgbytes(in,
tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0'; + tuple->values[i].len = len;The cursor should be set to 0 in the text formatted case too if this is
how we chose to encode data.However I am not quite convinced I like the StringInfoData usage here.
Why not just change the struct to include additional array of lengths
rather than replacing the existing values array with StringInfoData
array, that seems generally both simpler and should have smaller memory
footprint too, no?
Can you explain this a bit more? I don't really see a huge difference in
memory usage.
We still need length and the data copied into LogicalRepTupleData when
sending the data in binary, no?
We could also merge the binary and changed arrays into single char array
named something like format (as data can be either unchanged, binary or
text) and just reuse same identifiers we have in protocol.
This seems like a good idea.
Dave Cramer
Show quoted text
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Hi,
On 08/03/2020 00:18, Dave Cramer wrote:
On Fri, 6 Mar 2020 at 08:54, Petr Jelinek <petr@2ndquadrant.com
<mailto:petr@2ndquadrant.com>> wrote:Hi Dave,
On 29/02/2020 18:44, Dave Cramer wrote:
rebased and removed the catversion bump.
Looked into this and it generally seems okay, but I do have one
gripe here:+ tuple->values[i].data =
palloc(len + 1);
+ /* and data */ + + pq_copymsgbytes(in,tuple->values[i].data, len);
+ tuple->values[i].len = len; + tuple->values[i].cursor = 0; + tuple->values[i].maxlen = len; + /* not strictly necessarybut the docs say it is required */
+ tuple->values[i].data[len]
= '\0';
+ break; + } + case 't': /* textformatted value */
+ { + tuple->changed[i] = true; + int len = pq_getmsgint(in,4); /* read length */
/* and data */
- tuple->values[i] =palloc(len + 1);
- pq_copymsgbytes(in,
tuple->values[i], len);
- tuple->values[i][len] = '\0'; + tuple->values[i].data =palloc(len + 1);
+ pq_copymsgbytes(in,
tuple->values[i].data, len);
+ tuple->values[i].data[len]
= '\0';
+ tuple->values[i].len = len;
The cursor should be set to 0 in the text formatted case too if this is
how we chose to encode data.However I am not quite convinced I like the StringInfoData usage here.
Why not just change the struct to include additional array of lengths
rather than replacing the existing values array with StringInfoData
array, that seems generally both simpler and should have smaller memory
footprint too, no?Can you explain this a bit more? I don't really see a huge difference in
memory usage.
We still need length and the data copied into LogicalRepTupleData when
sending the data in binary, no?
Yes but we don't need to have fixed sized array of 1600 elements that
contains maxlen and cursor positions of the StringInfoData struct which
we don't use for anything AFAICS.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
On Fri, 3 Apr 2020 at 03:43, Petr Jelinek <petr@2ndquadrant.com> wrote:
Hi,
On 08/03/2020 00:18, Dave Cramer wrote:
On Fri, 6 Mar 2020 at 08:54, Petr Jelinek <petr@2ndquadrant.com
<mailto:petr@2ndquadrant.com>> wrote:Hi Dave,
On 29/02/2020 18:44, Dave Cramer wrote:
rebased and removed the catversion bump.
Looked into this and it generally seems okay, but I do have one
gripe here:+ tuple->values[i].data =
palloc(len + 1);
+ /* and data */ + + pq_copymsgbytes(in,tuple->values[i].data, len);
+ tuple->values[i].len = len; + tuple->values[i].cursor = 0; + tuple->values[i].maxlen =len;
+ /* not strictly necessary
but the docs say it is required */
+ tuple->values[i].data[len]
= '\0';
+ break; + } + case 't': /* textformatted value */
+ { + tuple->changed[i] = true; + int len = pq_getmsgint(in,4); /* read length */
/* and data */
- tuple->values[i] =palloc(len + 1);
- pq_copymsgbytes(in,
tuple->values[i], len);
- tuple->values[i][len] =
'\0';
+ tuple->values[i].data =
palloc(len + 1);
+ pq_copymsgbytes(in,
tuple->values[i].data, len);
+ tuple->values[i].data[len]
= '\0';
+ tuple->values[i].len = len;
The cursor should be set to 0 in the text formatted case too if this
is
how we chose to encode data.
However I am not quite convinced I like the StringInfoData usage
here.
Why not just change the struct to include additional array of lengths
rather than replacing the existing values array with StringInfoData
array, that seems generally both simpler and should have smallermemory
footprint too, no?
Can you explain this a bit more? I don't really see a huge difference in
memory usage.
We still need length and the data copied into LogicalRepTupleData when
sending the data in binary, no?Yes but we don't need to have fixed sized array of 1600 elements that
contains maxlen and cursor positions of the StringInfoData struct which
we don't use for anything AFAICS.
OK, I can see an easy way to only allocate the number of elements required
but since OidReceiveFunctionCall takes
StringInfo as one of it's arguments it seems like an easy path unless there
is something I am missing ?
Dave
On Fri, 3 Apr 2020 at 16:44, Dave Cramer <davecramer@gmail.com> wrote:
On Fri, 3 Apr 2020 at 03:43, Petr Jelinek <petr@2ndquadrant.com> wrote:
Hi,
On 08/03/2020 00:18, Dave Cramer wrote:
On Fri, 6 Mar 2020 at 08:54, Petr Jelinek <petr@2ndquadrant.com
<mailto:petr@2ndquadrant.com>> wrote:Hi Dave,
On 29/02/2020 18:44, Dave Cramer wrote:
rebased and removed the catversion bump.
Looked into this and it generally seems okay, but I do have one
gripe here:+ tuple->values[i].data =
palloc(len + 1);
+ /* and data */ + + pq_copymsgbytes(in,tuple->values[i].data, len);
+ tuple->values[i].len = len; + tuple->values[i].cursor =0;
+ tuple->values[i].maxlen =
len;
+ /* not strictly necessary
but the docs say it is required */
+ tuple->values[i].data[len]
= '\0';
+ break; + } + case 't': /* textformatted value */
+ { + tuple->changed[i] = true; + int len = pq_getmsgint(in,4); /* read length */
/* and data */
- tuple->values[i] =palloc(len + 1);
- pq_copymsgbytes(in,
tuple->values[i], len);
- tuple->values[i][len] =
'\0';
+ tuple->values[i].data =
palloc(len + 1);
+ pq_copymsgbytes(in,
tuple->values[i].data, len);
+ tuple->values[i].data[len]
= '\0';
+ tuple->values[i].len = len;
The cursor should be set to 0 in the text formatted case too if
this is
how we chose to encode data.
However I am not quite convinced I like the StringInfoData usage
here.
Why not just change the struct to include additional array of
lengths
rather than replacing the existing values array with StringInfoData
array, that seems generally both simpler and should have smallermemory
footprint too, no?
Can you explain this a bit more? I don't really see a huge difference
in
memory usage.
We still need length and the data copied into LogicalRepTupleData when
sending the data in binary, no?Yes but we don't need to have fixed sized array of 1600 elements that
contains maxlen and cursor positions of the StringInfoData struct which
we don't use for anything AFAICS.
New patch that fixes a number of errors in the check for validity as well
as reduces the memory usage by
dynamically allocating the data changed as well as collapsing the changed
and binary arrays into a format array.
Dave Cramer
Show quoted text
Attachments:
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From 7b1824d9ad6069343b1204ee72b5e5435cf0bdce Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 11 Jun 2019 15:35:11 -0400
Subject: [PATCH 1/8] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 ++++++++++-------
src/backend/replication/logical/worker.c | 124 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 34 +++++-
src/include/replication/logicalproto.h | 20 ++--
src/include/replication/pgoutput.h | 1 +
5 files changed, 201 insertions(+), 86 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..935b167ec2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -155,14 +157,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -171,7 +169,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -179,7 +176,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +193,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +234,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +256,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +264,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +276,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +424,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +440,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +476,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -514,6 +521,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -524,25 +534,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 673ebd211d..049a3b3392 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -310,13 +310,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -342,18 +341,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -387,11 +408,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -429,21 +456,43 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -609,8 +658,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -632,7 +685,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -720,9 +773,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -763,8 +819,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
Assert(rel->localrel->rd_rel->relkind == RELKIND_RELATION);
@@ -814,8 +870,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -858,8 +913,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -883,7 +941,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
Assert(rel->localrel->rd_rel->relkind == RELKIND_RELATION);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 552a70cffa..2b38e6375e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -92,11 +92,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -142,6 +146,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -176,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -348,7 +369,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_insert(ctx->out, relation,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
case REORDER_BUFFER_CHANGE_UPDATE:
@@ -358,7 +380,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_update(ctx->out, relation, oldtuple,
- &change->data.tp.newtuple->tuple);
+ &change->data.tp.newtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -367,7 +390,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_delete(ctx->out, relation,
- &change->data.tp.oldtuple->tuple);
+ &change->data.tp.oldtuple->tuple,
+ data->binary_basetypes);
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..ac2e17a769 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -87,16 +91,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From 190f43a30517ddb0b474052da9860b3fb427b22f Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Fri, 14 Jun 2019 15:39:47 -0400
Subject: [PATCH 2/8] add binary column to pg_subscription support create and
alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 ++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/catversion.h | 58 -------------------
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
7 files changed, 42 insertions(+), 66 deletions(-)
delete mode 100644 src/include/catalog/catversion.h
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 813ea8bfc3..0bee6728d6 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1122,7 +1122,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7f156673f7..e0ea6cd413 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..2a9d966ec1 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -402,6 +402,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -423,6 +424,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 049a3b3392..c10414b16d 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1866,6 +1866,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h
deleted file mode 100644
index 6d91fa2bde..0000000000
--- a/src/include/catalog/catversion.h
+++ /dev/null
@@ -1,58 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * catversion.h
- * "Catalog version number" for PostgreSQL.
- *
- * The catalog version number is used to flag incompatible changes in
- * the PostgreSQL system catalogs. Whenever anyone changes the format of
- * a system catalog relation, or adds, deletes, or modifies standard
- * catalog entries in such a way that an updated backend wouldn't work
- * with an old database (or vice versa), the catalog version number
- * should be changed. The version number stored in pg_control by initdb
- * is checked against the version number compiled into the backend at
- * startup time, so that a backend can refuse to run in an incompatible
- * database.
- *
- * The point of this feature is to provide a finer grain of compatibility
- * checking than is possible from looking at the major version number
- * stored in PG_VERSION. It shouldn't matter to end users, but during
- * development cycles we usually make quite a few incompatible changes
- * to the contents of the system catalogs, and we don't want to bump the
- * major version number for each one. What we can do instead is bump
- * this internal version number. This should save some grief for
- * developers who might otherwise waste time tracking down "bugs" that
- * are really just code-vs-database incompatibilities.
- *
- * The rule for developers is: if you commit a change that requires
- * an initdb, you should update the catalog version number (as well as
- * notifying the pgsql-hackers mailing list, which has been the
- * informal practice for a long time).
- *
- * The catalog version number is placed here since modifying files in
- * include/catalog is the most common kind of initdb-forcing change.
- * But it could be used to protect any kind of incompatible change in
- * database contents or layout, such as altering tuple headers.
- *
- *
- * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * src/include/catalog/catversion.h
- *
- *-------------------------------------------------------------------------
- */
-#ifndef CATVERSION_H
-#define CATVERSION_H
-
-/*
- * We could use anything we wanted for version numbers, but I recommend
- * following the "YYYYMMDDN" style often used for DNS zone serial numbers.
- * YYYYMMDD are the date of the change, and N is the number of the change
- * on that day. (Hopefully we'll never commit ten independent sets of
- * catalog changes on the same day...)
- */
-
-/* yyyymmddN */
-#define CATALOG_VERSION_NO 202004022
-
-#endif
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index cf3e43128c..13f38b34d6 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -168,6 +168,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 5dbb45664d2880c3aef64a2555ecce10e097c401 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 17 Jun 2019 10:26:59 -0400
Subject: [PATCH 3/8] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 64614b569c..381a7e730b 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6667,6 +6667,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>If true, the subscription is enabled and should be replicating.</entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry><structfield>subsynccommit</structfield></entry>
<entry><type>text</type></entry>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8beadbaa..66172a9b05 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..1059370a2c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From 52418a9a79b8f4913e13a75d921d4251eeaf1c5e Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 5 Nov 2019 07:14:30 -0500
Subject: [PATCH 5/8] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 6494b645d0..d4c6283a17 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -547,10 +547,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -565,8 +566,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From f4856757288be4a313f3561dece727a3f7cdb7cb Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Wed, 30 Oct 2019 09:58:59 -0400
Subject: [PATCH 4/8] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 935b167ec2..6494b645d0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -157,18 +157,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c10414b16d..58ada925ee 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -659,10 +659,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index ac2e17a769..85351c6093 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -92,7 +92,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From 5250b55a55275b5431e246feff5ca18fe4540176 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:37:35 -0500
Subject: [PATCH 6/8] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From d920c0637569d81ea343851715622318ad0e50f1 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 11 Nov 2019 14:38:08 -0500
Subject: [PATCH 7/8] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 2a9d966ec1..9130d645b6 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -402,7 +402,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -424,9 +423,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 2b38e6375e..fab6f75b49 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -98,10 +98,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -161,10 +176,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
0008-Changed-binary-and-changed-to-format-and-use-existin.patchapplication/octet-stream; name=0008-Changed-binary-and-changed-to-format-and-use-existin.patchDownload
From 223509362254d6962a336aeeb957b07f2d237a5c Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 7 Apr 2020 15:35:47 -0400
Subject: [PATCH 8/8] Changed binary and changed to format and use existing
codes to get rid of one array\ Dynamically allocate the arrays to avoid
allocating max columns sized arrays Fixed numerous issues with checking for
validity
---
.../libpqwalreceiver/libpqwalreceiver.c | 39 ++++++++-----
src/backend/replication/logical/proto.c | 56 +++++++++++--------
src/backend/replication/logical/worker.c | 30 +++++-----
src/backend/replication/pgoutput/pgoutput.c | 35 +++++++-----
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 8 +--
6 files changed, 99 insertions(+), 71 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9130d645b6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -425,36 +425,49 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
pfree(pubnames_str);
if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
- appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
- appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
- appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
- appendStringInfo(&cmd, ", bigendian %d",
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
#ifdef WORDS_BIGENDIAN
true
#else
false
#endif
);
- appendStringInfo(&cmd, ", float4_byval %d",
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
#ifdef USE_FLOAT4_BYVAL
- true
+ true
#else
- false
+ false
#endif
- );
- appendStringInfo(&cmd, ", float8_byval %d",
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
#ifdef USE_FLOAT8_BYVAL
- true
+ true
#else
- false
+ false
#endif
- );
- appendStringInfo(&cmd, ", integer_datetimes %d",
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
#ifdef USE_INTEGER_DATETIMES
true
#else
false
#endif
+#endif
+
);
}
appendStringInfoChar(&cmd, ')');
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index d4c6283a17..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -521,10 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
- /* default is text */
- memset(tuple->binary, 0, sizeof(tuple->binary));
/* Read the data */
for (i = 0; i < natts; i++)
@@ -536,45 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i].len = 0;
- tuple->values[i].data = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i].len = 0;
- tuple->values[i].data = NULL;
- break;
+ {
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
case 'b': /* binary formatted value */
{
int len;
- tuple->changed[i] = true;
- tuple->binary[i] = true;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
len = pq_getmsgint(in, 4); /* read length */
- tuple->values[i].data = palloc(len + 1);
+ value->data = palloc(len + 1);
/* and data */
- pq_copymsgbytes(in, tuple->values[i].data, len);
- tuple->values[i].len = len;
- tuple->values[i].cursor = 0;
- tuple->values[i].maxlen = len;
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
/* not strictly necessary but the docs say it is required */
- tuple->values[i].data[len] = '\0';
+ value->data[len] = '\0';
+ tuple->values[i] = value;
break;
}
case 't': /* text formatted value */
{
int len;
- tuple->changed[i] = true;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i].data = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i].data, len);
- tuple->values[i].data[len] = '\0';
- tuple->values[i].len = len;
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 58ada925ee..a46c97c188 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -341,28 +341,28 @@ slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- tupleData->values[remoteattnum].data != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->binary[remoteattnum])
+ if (tupleData->format[remoteattnum] == 'b')
{
Oid typreceive;
Oid typioparam;
getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
- int cursor = tupleData->values[remoteattnum].cursor;
+ int cursor = tupleData->values[remoteattnum]->cursor;
slot->tts_values[i] =
- OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
typioparam, att->atttypmod);
/*
* Do not advance the cursor in case we need to re-read this
* This saves us from pushing all of this type logic into proto.c
*/
- tupleData->values[remoteattnum].cursor = cursor;
+ tupleData->values[remoteattnum]->cursor = cursor;
}
else
@@ -372,7 +372,7 @@ slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
getTypeInputInfo(att->atttypid, &typinput, &typioparam);
slot->tts_values[i] =
- OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
typioparam, att->atttypmod);
}
slot->tts_isnull[i] = false;
@@ -456,16 +456,16 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!tupleData->changed[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (tupleData->values[remoteattnum].data != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->binary[remoteattnum])
+ if (tupleData->format[remoteattnum] == 'b')
{
Oid typreceive;
Oid typioparam;
@@ -473,15 +473,15 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
- int cursor = tupleData->values[remoteattnum].cursor;
+ int cursor = tupleData->values[remoteattnum]->cursor;
slot->tts_values[i] =
- OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
typioparam, att->atttypmod);
/*
* Do not advance the cursor in case we need to re-read this
* This saves us from pushing all of this type logic into proto.c
*/
- tupleData->values[remoteattnum].cursor = cursor;
+ tupleData->values[remoteattnum]->cursor = cursor;
}
else
{
@@ -490,7 +490,7 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
getTypeInputInfo(att->atttypid, &typinput, &typioparam);
slot->tts_values[i] =
- OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
typioparam, att->atttypmod);
}
slot->tts_isnull[i] = false;
@@ -808,7 +808,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -870,7 +870,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, rel, &newtup);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index fab6f75b49..050ccfaadb 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -131,7 +132,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -152,7 +153,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -167,7 +168,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (binary_option_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
binary_option_given = true;
if (!parse_bool(strVal(defel->arg), &parsed))
@@ -185,7 +186,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_datum_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
sizeof_datum_given = true;
if (!scanint8(strVal(defel->arg), true, &datum_size))
@@ -199,7 +200,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_int_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
sizeof_int_given = true;
if (!scanint8(strVal(defel->arg), true, &int_size))
@@ -213,8 +214,8 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_long_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
- sizeof_int_given = true;
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
if (!scanint8(strVal(defel->arg), true, &long_size))
ereport(ERROR,
@@ -227,7 +228,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (big_endian_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
big_endian_given = true;
if (!parse_bool(strVal(defel->arg), &bigendian))
@@ -241,7 +242,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float4_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
float4_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float4_byval))
@@ -255,7 +256,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float8_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
float8_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float8_byval))
@@ -263,13 +264,13 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid float8_byval option")));
}
- else if (strcmp(defel->defname, "integer_date_times") == 0)
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
{
if (integer_datetimes_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
integer_datetimes_given = true;
if (!parse_bool(strVal(defel->arg), &integer_datetimes))
@@ -312,17 +313,21 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible endianness")));
if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
#ifdef USE_FLOAT4_BYVAL
true
#else
false
+#endif
#endif
)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible float4_byval")));
if( float8_byval !=
-#ifdef USE_FLOAT4_BYVAL
+#ifdef USE_FLOAT8_BYVAL
true
#else
false
@@ -333,10 +338,14 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("incompatible float8_byval")));
if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
#ifdef USE_INTEGER_DATETIMES
true
#else
false
+#endif
#endif
)
ereport(ERROR,
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 32f739706d..3504a19728 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 85351c6093..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -31,13 +31,11 @@
typedef struct LogicalRepTupleData
{
/* column values */
- StringInfoData values[MaxTupleAttributeNumber];
+ StringInfoData **values;
- /* markers for binary */
- bool binary[MaxTupleAttributeNumber];
+ /* markers for changed/unchanged/binary/text */
+ char *format;
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
--
2.20.1 (Apple Git-117)
On 7 Apr 2020, at 21:45, Dave Cramer <davecramer@gmail.com> wrote:
New patch that fixes a number of errors in the check for validity as well as reduces the memory usage by
dynamically allocating the data changed as well as collapsing the changed and binary arrays into a format array.
The 0001 patch fails to apply, and possibly other in the series. Please submit
a rebased version. Marking the CF entry as Waiting for Author in the meantime.
cheers ./daniel
Honestly I'm getting a little weary of fixing this up only to have the
patch not get reviewed.
Apparently it has no value so unless someone is willing to review it and
get it committed I'm just going to let it go.
Thanks,
Dave Cramer
On Wed, 1 Jul 2020 at 04:53, Daniel Gustafsson <daniel@yesql.se> wrote:
Show quoted text
On 7 Apr 2020, at 21:45, Dave Cramer <davecramer@gmail.com> wrote:
New patch that fixes a number of errors in the check for validity as
well as reduces the memory usage by
dynamically allocating the data changed as well as collapsing the
changed and binary arrays into a format array.
The 0001 patch fails to apply, and possibly other in the series. Please
submit
a rebased version. Marking the CF entry as Waiting for Author in the
meantime.cheers ./daniel
rebased
Thanks,
Dave Cramer
On Wed, 1 Jul 2020 at 06:43, Dave Cramer <davecramer@gmail.com> wrote:
Show quoted text
Honestly I'm getting a little weary of fixing this up only to have the
patch not get reviewed.Apparently it has no value so unless someone is willing to review it and
get it committed I'm just going to let it go.Thanks,
Dave Cramer
On Wed, 1 Jul 2020 at 04:53, Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Apr 2020, at 21:45, Dave Cramer <davecramer@gmail.com> wrote:
New patch that fixes a number of errors in the check for validity as
well as reduces the memory usage by
dynamically allocating the data changed as well as collapsing the
changed and binary arrays into a format array.
The 0001 patch fails to apply, and possibly other in the series. Please
submit
a rebased version. Marking the CF entry as Waiting for Author in the
meantime.cheers ./daniel
Attachments:
0001-First-pass-at-working-code-without-subscription-opti.patchapplication/octet-stream; name=0001-First-pass-at-working-code-without-subscription-opti.patchDownload
From c508a4a3b9b181467605b902e950892058ad0c54 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:14:40 -0400
Subject: [PATCH 1/8] First pass at working code without subscription options
---
src/backend/replication/logical/proto.c | 108 +++++++++++------
src/backend/replication/logical/worker.c | 128 ++++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 33 ++++-
src/include/replication/logicalproto.h | 20 +--
src/include/replication/pgoutput.h | 1 +
5 files changed, 202 insertions(+), 88 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..935b167ec2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -155,14 +157,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
*
* Fills the new tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
@@ -171,7 +169,6 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
@@ -179,7 +176,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +193,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +234,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +256,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +264,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +276,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +424,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +440,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +476,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -514,6 +521,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
memset(tuple->changed, 0, sizeof(tuple->changed));
+ /* default is text */
+ memset(tuple->binary, 0, sizeof(tuple->binary));
+
/* Read the data */
for (i = 0; i < natts; i++)
{
@@ -524,25 +534,43 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
tuple->changed[i] = true;
break;
case 'u': /* unchanged column */
/* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
+ tuple->values[i].len = 0;
+ tuple->values[i].data = NULL;
break;
- case 't': /* text formatted value */
+ case 'b': /* binary formatted value */
{
- int len;
-
tuple->changed[i] = true;
+ tuple->binary[i] = true;
+
+ int len = pq_getmsgint(in, 4); /* read length */
- len = pq_getmsgint(in, 4); /* read length */
+ tuple->values[i].data = palloc(len + 1);
+ /* and data */
+
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].len = len;
+ tuple->values[i].cursor = 0;
+ tuple->values[i].maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ tuple->values[i].data[len] = '\0';
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ tuple->changed[i] = true;
+ int len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ tuple->values[i].data = palloc(len + 1);
+ pq_copymsgbytes(in, tuple->values[i].data, len);
+ tuple->values[i].data[len] = '\0';
+ tuple->values[i].len = len;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..c958c4d8a7 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -396,11 +417,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +465,43 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (!tupleData->changed[remoteattnum])
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum].data != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->binary[remoteattnum])
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+
+
+ int cursor = tupleData->values[remoteattnum].cursor;
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum].cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +667,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_insert(s, &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +694,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +786,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -776,8 +832,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +887,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, rel, &newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +930,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +958,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1154,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
+ slot_modify_data(remoteslot_part, localslot,
part_entry,
- newtup->values, newtup->changed);
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..5752731374 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -118,11 +118,15 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ // default to false
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -168,6 +172,22 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -202,7 +222,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +418,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +433,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +457,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +477,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..ac2e17a769 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,8 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData values[MaxTupleAttributeNumber];
+
+ /* markers for binary */
+ bool binary[MaxTupleAttributeNumber];
+
/* markers for changed/unchanged column values: */
bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
@@ -87,16 +91,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
--
2.20.1 (Apple Git-117)
0005-Remove-C99-declaration-and-code.patchapplication/octet-stream; name=0005-Remove-C99-declaration-and-code.patchDownload
From 3be52db105d78c0f305b2503176adcee78fed49c Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:17:38 -0400
Subject: [PATCH 5/8] Remove C99 declaration and code
---
src/backend/replication/logical/proto.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 6494b645d0..d4c6283a17 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -547,10 +547,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
break;
case 'b': /* binary formatted value */
{
+ int len;
tuple->changed[i] = true;
tuple->binary[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
tuple->values[i].data = palloc(len + 1);
/* and data */
@@ -565,8 +566,9 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
}
case 't': /* text formatted value */
{
+ int len;
tuple->changed[i] = true;
- int len = pq_getmsgint(in, 4); /* read length */
+ len = pq_getmsgint(in, 4); /* read length */
/* and data */
tuple->values[i].data = palloc(len + 1);
--
2.20.1 (Apple Git-117)
0002-add-binary-column-to-pg_subscription.patchapplication/octet-stream; name=0002-add-binary-column-to-pg_subscription.patchDownload
From 9ad3dda95b96137232aef4be7632389921830892 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:15:17 -0400
Subject: [PATCH 2/8] add binary column to pg_subscription support create and
alter subcription with binary option
---
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 +++++++++++++++----
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/include/catalog/pg_subscription.h | 4 ++
src/include/replication/walreceiver.h | 1 +
6 files changed, 42 insertions(+), 8 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5314e9348f..9a853fe48d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..f2590ff565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..2a9d966ec1 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -402,6 +402,7 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
+ bool binary;
appendStringInfoString(&cmd, " (");
@@ -423,6 +424,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary)
+ appendStringInfo(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c958c4d8a7..b46e8991cd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -2164,6 +2164,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..1ee316f6d7 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0004-get-relid-inside-of-logical_read_insert.patchapplication/octet-stream; name=0004-get-relid-inside-of-logical_read_insert.patchDownload
From 478b1e5e54ef85cc98f62037b88b18b23a5e7785 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:17:06 -0400
Subject: [PATCH 4/8] get relid inside of logical_read_insert
---
src/backend/replication/logical/proto.c | 6 ++++--
src/backend/replication/logical/worker.c | 4 ++--
src/include/replication/logicalproto.h | 2 +-
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 935b167ec2..6494b645d0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -157,18 +157,20 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool b
*
* Fills the new tuple.
*/
-void
+LogicalRepRelId
logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
{
char action;
+ LogicalRepRelId relid;
+ relid = pq_getmsgint(in, 4);
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
+ return relid;
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index b46e8991cd..fadfd1dfaf 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -668,10 +668,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
/* read the relation id */
- relid = pq_getmsgint(s, 4);
+ relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
- logicalrep_read_insert(s, &newtup);
+
if (!should_apply_changes_for_rel(rel))
{
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index ac2e17a769..85351c6093 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -92,7 +92,7 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
HeapTuple newtuple, bool binary_basetypes);
-extern void logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
+extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
HeapTuple newtuple, bool binary_basetypes);
extern void logicalrep_read_update(StringInfo in,
--
2.20.1 (Apple Git-117)
0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0003-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From e5f29479d61b4eaa1df3a813b27d0b2db9aff18d Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:16:17 -0400
Subject: [PATCH 3/8] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 49a881b262..2169fad920 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7503,6 +7503,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..ab712910b2 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..770835edde 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0006-make-sure-the-subcription-is-marked-as-binary.patchapplication/octet-stream; name=0006-make-sure-the-subcription-is-marked-as-binary.patchDownload
From 9b57d34bcfc516eed2453d0c03f075e6d44c4cdb Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:18:05 -0400
Subject: [PATCH 6/8] make sure the subcription is marked as binary
---
src/backend/catalog/pg_subscription.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
--
2.20.1 (Apple Git-117)
0007-check-that-the-subscriber-is-compatible-with-the-pub.patchapplication/octet-stream; name=0007-check-that-the-subscriber-is-compatible-with-the-pub.patchDownload
From e791d3c501523e569b5893377a8c7f5af96fb57b Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:18:40 -0400
Subject: [PATCH 7/8] check that the subscriber is compatible with the
publisher
---
.../libpqwalreceiver/libpqwalreceiver.c | 36 +++-
src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++++
2 files changed, 212 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 2a9d966ec1..9130d645b6 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -402,7 +402,6 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
char *pubnames_str;
List *pubnames;
char *pubnames_literal;
- bool binary;
appendStringInfoString(&cmd, " (");
@@ -424,9 +423,40 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary)
+ if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
-
+ appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian %d",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval %d",
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval %d",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes %d",
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5752731374..70c9008af9 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -124,10 +124,25 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool protocol_version_given = false;
bool publication_names_given = false;
bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
// default to false
*binary_basetypes = false;
+
foreach(lc, options)
{
DefElem *defel = (DefElem *) lfirst(lc);
@@ -187,10 +202,174 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("invalid binary option")));
*binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_date_times") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
}
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
--
2.20.1 (Apple Git-117)
0008-Changed-binary-and-changed-to-format-and-use-existin.patchapplication/octet-stream; name=0008-Changed-binary-and-changed-to-format-and-use-existin.patchDownload
From d3113e2f43b4a99b53ceceaedffd0940805f66ba Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Thu, 2 Jul 2020 12:18:59 -0400
Subject: [PATCH 8/8] Changed binary and changed to format and use existing
codes to get rid of one array Dynamically allocate the arrays to avoid
allocating max columns sized arrays Fixed numerous issues with checking for
validity
---
.../libpqwalreceiver/libpqwalreceiver.c | 39 ++++++++-----
src/backend/replication/logical/proto.c | 56 +++++++++++--------
src/backend/replication/logical/worker.c | 30 +++++-----
src/backend/replication/pgoutput/pgoutput.c | 35 +++++++-----
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 8 +--
6 files changed, 99 insertions(+), 71 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 9130d645b6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -425,36 +425,49 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
pfree(pubnames_str);
if (options->proto.logical.binary) {
appendStringInfo(&cmd, ", binary 'true'");
- appendStringInfo(&cmd, ", sizeof_datum %zu", sizeof(Datum));
- appendStringInfo(&cmd, ", sizeof_int %zu", sizeof(int));
- appendStringInfo(&cmd, ", sizeof_long %zu", sizeof(long));
- appendStringInfo(&cmd, ", bigendian %d",
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
#ifdef WORDS_BIGENDIAN
true
#else
false
#endif
);
- appendStringInfo(&cmd, ", float4_byval %d",
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
#ifdef USE_FLOAT4_BYVAL
- true
+ true
#else
- false
+ false
#endif
- );
- appendStringInfo(&cmd, ", float8_byval %d",
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
#ifdef USE_FLOAT8_BYVAL
- true
+ true
#else
- false
+ false
#endif
- );
- appendStringInfo(&cmd, ", integer_datetimes %d",
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
#ifdef USE_INTEGER_DATETIMES
true
#else
false
#endif
+#endif
+
);
}
appendStringInfoChar(&cmd, ')');
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index d4c6283a17..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -521,10 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
- /* default is text */
- memset(tuple->binary, 0, sizeof(tuple->binary));
/* Read the data */
for (i = 0; i < natts; i++)
@@ -536,45 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i].len = 0;
- tuple->values[i].data = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i].len = 0;
- tuple->values[i].data = NULL;
- break;
+ {
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
case 'b': /* binary formatted value */
{
int len;
- tuple->changed[i] = true;
- tuple->binary[i] = true;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
len = pq_getmsgint(in, 4); /* read length */
- tuple->values[i].data = palloc(len + 1);
+ value->data = palloc(len + 1);
/* and data */
- pq_copymsgbytes(in, tuple->values[i].data, len);
- tuple->values[i].len = len;
- tuple->values[i].cursor = 0;
- tuple->values[i].maxlen = len;
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
/* not strictly necessary but the docs say it is required */
- tuple->values[i].data[len] = '\0';
+ value->data[len] = '\0';
+ tuple->values[i] = value;
break;
}
case 't': /* text formatted value */
{
int len;
- tuple->changed[i] = true;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i].data = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i].data, len);
- tuple->values[i].data[len] = '\0';
- tuple->values[i].len = len;
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fadfd1dfaf..261d017fc0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -350,28 +350,28 @@ slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- tupleData->values[remoteattnum].data != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->binary[remoteattnum])
+ if (tupleData->format[remoteattnum] == 'b')
{
Oid typreceive;
Oid typioparam;
getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
- int cursor = tupleData->values[remoteattnum].cursor;
+ int cursor = tupleData->values[remoteattnum]->cursor;
slot->tts_values[i] =
- OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
typioparam, att->atttypmod);
/*
* Do not advance the cursor in case we need to re-read this
* This saves us from pushing all of this type logic into proto.c
*/
- tupleData->values[remoteattnum].cursor = cursor;
+ tupleData->values[remoteattnum]->cursor = cursor;
}
else
@@ -381,7 +381,7 @@ slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
getTypeInputInfo(att->atttypid, &typinput, &typioparam);
slot->tts_values[i] =
- OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
typioparam, att->atttypmod);
}
slot->tts_isnull[i] = false;
@@ -465,16 +465,16 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!tupleData->changed[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (tupleData->values[remoteattnum].data != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->binary[remoteattnum])
+ if (tupleData->format[remoteattnum] == 'b')
{
Oid typreceive;
Oid typioparam;
@@ -482,15 +482,15 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
- int cursor = tupleData->values[remoteattnum].cursor;
+ int cursor = tupleData->values[remoteattnum]->cursor;
slot->tts_values[i] =
- OidReceiveFunctionCall(typreceive, &tupleData->values[remoteattnum],
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
typioparam, att->atttypmod);
/*
* Do not advance the cursor in case we need to re-read this
* This saves us from pushing all of this type logic into proto.c
*/
- tupleData->values[remoteattnum].cursor = cursor;
+ tupleData->values[remoteattnum]->cursor = cursor;
}
else
{
@@ -499,7 +499,7 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
getTypeInputInfo(att->atttypid, &typinput, &typioparam);
slot->tts_values[i] =
- OidInputFunctionCall(typinput, tupleData->values[remoteattnum].data,
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
typioparam, att->atttypmod);
}
slot->tts_isnull[i] = false;
@@ -821,7 +821,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -887,7 +887,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_data(remoteslot, localslot, rel, &newtup);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 70c9008af9..65c2e5d658 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -157,7 +158,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -178,7 +179,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -193,7 +194,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (binary_option_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
binary_option_given = true;
if (!parse_bool(strVal(defel->arg), &parsed))
@@ -211,7 +212,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_datum_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
sizeof_datum_given = true;
if (!scanint8(strVal(defel->arg), true, &datum_size))
@@ -225,7 +226,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_int_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
sizeof_int_given = true;
if (!scanint8(strVal(defel->arg), true, &int_size))
@@ -239,8 +240,8 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_long_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
- sizeof_int_given = true;
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
if (!scanint8(strVal(defel->arg), true, &long_size))
ereport(ERROR,
@@ -253,7 +254,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (big_endian_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
big_endian_given = true;
if (!parse_bool(strVal(defel->arg), &bigendian))
@@ -267,7 +268,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float4_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
float4_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float4_byval))
@@ -281,7 +282,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float8_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
float8_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float8_byval))
@@ -289,13 +290,13 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid float8_byval option")));
}
- else if (strcmp(defel->defname, "integer_date_times") == 0)
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
{
if (integer_datetimes_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
integer_datetimes_given = true;
if (!parse_bool(strVal(defel->arg), &integer_datetimes))
@@ -338,17 +339,21 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible endianness")));
if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
#ifdef USE_FLOAT4_BYVAL
true
#else
false
+#endif
#endif
)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible float4_byval")));
if( float8_byval !=
-#ifdef USE_FLOAT4_BYVAL
+#ifdef USE_FLOAT8_BYVAL
true
#else
false
@@ -359,10 +364,14 @@ parse_output_parameters(List *options, uint32 *protocol_version,
errmsg("incompatible float8_byval")));
if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
#ifdef USE_INTEGER_DATETIMES
true
#else
false
+#endif
#endif
)
ereport(ERROR,
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 8f3ec6bde1..ec4fa01f30 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 85351c6093..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -31,13 +31,11 @@
typedef struct LogicalRepTupleData
{
/* column values */
- StringInfoData values[MaxTupleAttributeNumber];
+ StringInfoData **values;
- /* markers for binary */
- bool binary[MaxTupleAttributeNumber];
+ /* markers for changed/unchanged/binary/text */
+ char *format;
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
--
2.20.1 (Apple Git-117)
On 2 Jul 2020, at 18:41, Dave Cramer <davecramer@gmail.com> wrote:
rebased
Thanks! The new version of 0001 patch has a compiler warning due to mixed
declarations and code:
worker.c: In function ‘slot_store_data’:
worker.c:366:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
int cursor = tupleData->values[remoteattnum]->cursor;
^
worker.c: In function ‘slot_modify_data’:
worker.c:485:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
int cursor = tupleData->values[remoteattnum]->cursor;
^
I didn't investigate to see if it was new, but Travis is running with Werror
which fails this build.
cheers ./daniel
On 2020-Jul-05, Daniel Gustafsson wrote:
On 2 Jul 2020, at 18:41, Dave Cramer <davecramer@gmail.com> wrote:
rebased
Thanks! The new version of 0001 patch has a compiler warning due to mixed
declarations and code:worker.c: In function ‘slot_store_data’:
worker.c:366:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
AFAICS this is fixed in 0005. I'm going to suggest to use "git rebase
-i" so that fixes for bugs that earlier patches introduce are applied as
fix-ups in those patches; we don't need or want to see the submitter's
intermediate versions. Ideally, each submitted patch should be free of
such problems, so that we can consider each individual patch in the
series in isolation. Indeed, evidently the cfbot consider things that
way.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 5 Jul 2020, at 23:11, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
On 2020-Jul-05, Daniel Gustafsson wrote:
On 2 Jul 2020, at 18:41, Dave Cramer <davecramer@gmail.com> wrote:
rebased
Thanks! The new version of 0001 patch has a compiler warning due to mixed
declarations and code:worker.c: In function ‘slot_store_data’:
worker.c:366:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]AFAICS this is fixed in 0005.
Yes and no, 0005 fixes one such instance but the one failing the build is
another one in worker.c (the below being from 0008 which in turn change the row
in question from previous patches):
+ int cursor = tupleData->values[remoteattnum]->cursor;
I'm going to suggest to use "git rebase -i"
+1
cheers ./daniel
On Sun, 5 Jul 2020 at 17:28, Daniel Gustafsson <daniel@yesql.se> wrote:
On 5 Jul 2020, at 23:11, Alvaro Herrera <alvherre@2ndquadrant.com>
wrote:
On 2020-Jul-05, Daniel Gustafsson wrote:
On 2 Jul 2020, at 18:41, Dave Cramer <davecramer@gmail.com> wrote:
rebased
Thanks! The new version of 0001 patch has a compiler warning due to
mixed
declarations and code:
worker.c: In function ‘slot_store_data’:
worker.c:366:5: error: ISO C90 forbids mixed declarations and code[-Werror=declaration-after-statement]
AFAICS this is fixed in 0005.
Yes and no, 0005 fixes one such instance but the one failing the build is
another one in worker.c (the below being from 0008 which in turn change
the row
in question from previous patches):+ int cursor = tupleData->values[remoteattnum]->cursor;
I'm going to suggest to use "git rebase -i"
+1
Strangely I don't see those errors when I build on my machine, but I will
fix them
as far as rebase -i do what is advised here for squashing them. Just one
patch now ?
Thanks,
On 6 Jul 2020, at 14:58, Dave Cramer <davecramer@gmail.com> wrote:
as far as rebase -i do what is advised here for squashing them. Just one patch now ?
One patch per logical change, if there are two disjoint changes in the patchset
where one builds on top of the other then multiple patches are of course fine.
My personal rule-of-thumb is to split it the way I envision it committed.
cheers ./daniel
On Mon, 6 Jul 2020 at 09:03, Daniel Gustafsson <daniel@yesql.se> wrote:
On 6 Jul 2020, at 14:58, Dave Cramer <davecramer@gmail.com> wrote:
as far as rebase -i do what is advised here for squashing them. Just one
patch now ?
One patch per logical change, if there are two disjoint changes in the
patchset
where one builds on top of the other then multiple patches are of course
fine.
My personal rule-of-thumb is to split it the way I envision it committed.
At this point it is the result of 3 rebases. I guess I'll have to break it
up differently..
Thanks,
Dave
On Mon, 6 Jul 2020 at 09:35, Dave Cramer <davecramer@gmail.com> wrote:
On Mon, 6 Jul 2020 at 09:03, Daniel Gustafsson <daniel@yesql.se> wrote:
On 6 Jul 2020, at 14:58, Dave Cramer <davecramer@gmail.com> wrote:
as far as rebase -i do what is advised here for squashing them. Just
one patch now ?
One patch per logical change, if there are two disjoint changes in the
patchset
where one builds on top of the other then multiple patches are of course
fine.
My personal rule-of-thumb is to split it the way I envision it committed.At this point it is the result of 3 rebases. I guess I'll have to break it
up differently..
OK, rebased it down to 2 patches, attached.
Show quoted text
Thanks,
Dave
Attachments:
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patchDownload
From b109ee7d04483b053676755223ba0ccb1b53c511 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:50:21 -0400
Subject: [PATCH 1/2] Add binary protocol for publications and subscriptions
for base types add binary column to pg_subscription support create and alter
subcription with binary option
check that the subscriber is compatible with the
publisher
Removed the array representing binary types in favour of
an array representing the format of each type and use existing codes 't' for
text, 'b' for binary and 'u' for unchanged.
Dynamically allocate the arrays to avoid allocating max columns sized arrays.
Fixed numerous issues with checking for validity
---
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 ++-
.../libpqwalreceiver/libpqwalreceiver.c | 46 ++++
src/backend/replication/logical/proto.c | 122 ++++++----
src/backend/replication/logical/worker.c | 128 +++++++---
src/backend/replication/pgoutput/pgoutput.c | 225 +++++++++++++++++-
src/include/catalog/pg_subscription.h | 4 +
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 20 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
12 files changed, 490 insertions(+), 101 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5314e9348f..9a853fe48d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..f2590ff565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,53 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary) {
+ appendStringInfo(&cmd, ", binary 'true'");
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -161,16 +163,13 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
return relid;
}
@@ -179,7 +178,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +195,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +236,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +258,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +278,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +426,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +442,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +478,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
+
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
{
- int len;
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
+ case 'b': /* binary formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ value->data = palloc(len + 1);
+ /* and data */
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..806e0a8bed 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -396,11 +417,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +465,42 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +666,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +693,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +785,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +820,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +831,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +886,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +929,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +957,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1153,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
+ slot_modify_data(remoteslot_part, localslot,
part_entry,
- newtup->values, newtup->changed);
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2163,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..65c2e5d658 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,30 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
+
+ // default to false
+ *binary_basetypes = false;
+
foreach(lc, options)
{
@@ -138,7 +158,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +179,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,9 +188,197 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
@@ -202,7 +410,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +606,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +621,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +645,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +665,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 8f3ec6bde1..ec4fa01f30 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,10 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,16 +89,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary_basetypes);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..1ee316f6d7 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 83672308a8c37e311c34f2c2f62d61c6b07aadd0 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:53:33 -0400
Subject: [PATCH 2/2] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 003d278370..27b2de6fc7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7503,6 +7503,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..ab712910b2 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..770835edde 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
On 7 Jul 2020, at 02:16, Dave Cramer <davecramer@gmail.com> wrote:
OK, rebased it down to 2 patches, attached.
I took a look at this patchset today. The feature clearly seems like something
which we'd benefit from having, especially if it allows for the kind of
extensions that were discussed at the beginning of this thread. In general I
think it's in pretty good shape, there are however a few comments:
The patch lacks any kind of test, which I think is required for it to be
considered for committing. It also doesn't update the \dRs view in psql to
include the subbinary column which IMO it should. I took the liberty to write
this as well as tests as I was playing with the patch, the attached 0003
contains this, while 0001 and 0002 are your patches included to ensure the
CFBot can do it's thing. This was kind of thrown together to have something
while testing, so it definately need a once-over or two.
The comment here implies that unchanged is the default value for format, but
isn't this actually setting it to text formatted value?
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
Also, since the values member isn't memset() with a default, this seems a bit
misleading at best no?
For the rest of the backend we aren't including the defname in the errmsg like
what is done here. Maybe we should, but I think that should be done
consistently if so, and we should stick to just "conflicting or redundant
options" for now. At the very least, this needs a comma between "options" and
the defname and ideally the defname wrapped in \".
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
These types of constructs are IMHO quite hard to read:
+ if(
+ #ifdef WORDS_BIGENDIAN
+ true
+ #else
+ false
+ #endif
+ != bigendian)
How about spelling out the statement completely for both cases, or perhaps
encapsulating the logic in a macro? Something like the below perhaps?
+ #ifdef WORDS_BIGENDIAN
+ if (bigendian != true)
+ #else
+ if (bigendian != false)
+ #endif
This change needs to be removed before a commit, just highlighting that here to
avoid it going unnoticed.
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
Reading this I'm left wondering if we shoulnd't introduce macros for the kinds,
since we now compare with 'u', 't' etc in quite a few places and add comments
explaining the types everywhere. A descriptive name would make it easier to
grep for all occurrences, and avoid the need for the comment lines. Thats not
necesarily for this patch though, but an observation from reading it.
I found a few smaller nitpicks as well, some of these might go away by a
pg_indent run but I've included them all here regardless:
This change, and the subsequent whitespace removal later in the same function,
seems a bit pointless:
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-
Braces should go on the next line:
+ if (options->proto.logical.binary) {
This should be a C /* ... */ comment, or perhaps just removed since the code
is quite self explanatory:
+ // default to false
+ *binary_basetypes = false;
Indentation here:
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
..as well as here (there are a few like this one):
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
Capitalization of "after" to make it a proper sentence:
+ * after we know that the subscriber is requesting binary check to make sure
Excessive whitespace and indentation in a few places, and not enough in some:
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
...
+ if ( *binary_basetypes == true )
...
+ if (sizeof(int) != int_size)
...
+ if( float4_byval !=
...
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
...
+ if (tupleData->format[remoteattnum] =='u')
...
+ bool binary_basetypes;
That's all for now.
cheers ./daniel
Attachments:
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patch; x-unix-mode=0644Download
From b109ee7d04483b053676755223ba0ccb1b53c511 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:50:21 -0400
Subject: [PATCH 1/2] Add binary protocol for publications and subscriptions
for base types add binary column to pg_subscription support create and alter
subcription with binary option
check that the subscriber is compatible with the
publisher
Removed the array representing binary types in favour of
an array representing the format of each type and use existing codes 't' for
text, 'b' for binary and 'u' for unchanged.
Dynamically allocate the arrays to avoid allocating max columns sized arrays.
Fixed numerous issues with checking for validity
---
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 ++-
.../libpqwalreceiver/libpqwalreceiver.c | 46 ++++
src/backend/replication/logical/proto.c | 122 ++++++----
src/backend/replication/logical/worker.c | 128 +++++++---
src/backend/replication/pgoutput/pgoutput.c | 225 +++++++++++++++++-
src/include/catalog/pg_subscription.h | 4 +
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 20 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
12 files changed, 490 insertions(+), 101 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5314e9348f..9a853fe48d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..f2590ff565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,53 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary) {
+ appendStringInfo(&cmd, ", binary 'true'");
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -161,16 +163,13 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
return relid;
}
@@ -179,7 +178,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +195,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +236,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +258,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +278,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +426,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +442,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +478,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
+
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
{
- int len;
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
+ case 'b': /* binary formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ value->data = palloc(len + 1);
+ /* and data */
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..806e0a8bed 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -396,11 +417,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +465,42 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +666,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +693,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +785,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +820,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +831,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +886,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +929,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +957,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1153,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
+ slot_modify_data(remoteslot_part, localslot,
part_entry,
- newtup->values, newtup->changed);
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2163,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..65c2e5d658 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,30 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
+
+ // default to false
+ *binary_basetypes = false;
+
foreach(lc, options)
{
@@ -138,7 +158,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +179,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,9 +188,197 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
@@ -202,7 +410,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +606,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +621,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +645,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +665,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 8f3ec6bde1..ec4fa01f30 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,10 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,16 +89,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary_basetypes);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..1ee316f6d7 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patch; x-unix-mode=0644Download
From 83672308a8c37e311c34f2c2f62d61c6b07aadd0 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:53:33 -0400
Subject: [PATCH 2/2] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 003d278370..27b2de6fc7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7503,6 +7503,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..ab712910b2 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..770835edde 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0003-Add-psql-support-and-tests.patchapplication/octet-stream; name=0003-Add-psql-support-and-tests.patch; x-unix-mode=0644Download
From dbe82ef21c370143cf47b3f2dd8392c44b12b0b8 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 7 Jul 2020 15:53:35 +0200
Subject: [PATCH 2/2] Add psql support and tests
---
src/bin/psql/describe.c | 8 +-
src/test/regress/expected/subscription.out | 56 +++++++----
src/test/regress/sql/subscription.sql | 12 +++
src/test/subscription/t/014_binary.pl | 108 +++++++++++++++++++++
4 files changed, 163 insertions(+), 21 deletions(-)
create mode 100644 src/test/subscription/t/014_binary.pl
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index cd39b913cd..485c3c6e7c 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5963,7 +5963,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5987,6 +5987,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Enabled"),
gettext_noop("Publication"));
+ /* Binary mode is only supported in v14 and higher */
+ if (pset.sversion >= 140000)
+ appendPQExpBuffer(&buf,
+ ", subbinary AS \"%s\"\n",
+ gettext_noop("Binary"));
+
if (verbose)
{
appendPQExpBuffer(&buf,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index e7add9d2b8..af6ed982ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -91,27 +91,27 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | t | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | t | {testpub2,testpub3} | f
(1 row)
ALTER SUBSCRIPTION regress_testsub DISABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f
(1 row)
COMMIT;
@@ -126,10 +126,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -155,6 +155,22 @@ DROP SUBSCRIPTION IF EXISTS regress_testsub;
NOTICE: subscription "regress_testsub" does not exist, skipping
DROP SUBSCRIPTION regress_testsub; -- fail
ERROR: subscription "regress_testsub" does not exist
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+ERROR: binary requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 9e234ab8b3..835bd05721 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -117,6 +117,18 @@ COMMIT;
DROP SUBSCRIPTION IF EXISTS regress_testsub;
DROP SUBSCRIPTION regress_testsub; -- fail
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/014_binary.pl b/src/test/subscription/t/014_binary.pl
new file mode 100644
index 0000000000..538114a7fc
--- /dev/null
+++ b/src/test/subscription/t/014_binary.pl
@@ -0,0 +1,108 @@
+# Binary mode logical replication test
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Create and initialize a publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create and initialize subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+my $ddl = qq(
+ CREATE TABLE public.test_numerical (
+ a INTEGER PRIMARY KEY,
+ b NUMERIC,
+ c FLOAT,
+ d BIGINT
+ );
+ CREATE TABLE public.test_arrays (
+ a INTEGER[] PRIMARY KEY,
+ b NUMERIC[],
+ c TEXT[]
+ ););
+
+# Create content on both sides of the replication
+$node_publisher->safe_psql('postgres', $ddl);
+$node_subscriber->safe_psql('postgres', $ddl);
+
+# Configure logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tpub FOR ALL TABLES");
+
+my $publisher_connstring = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tsub CONNECTION '$publisher_connstring' " .
+ "PUBLICATION tpub WITH (slot_name = tpub_slot, binary = true)");
+
+# Ensure nodes are in sync with eachother
+$node_publisher->wait_for_catchup('tsub');
+$node_subscriber->poll_query_until('postgres',
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');")
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Insert some content and make sure it's replicated across
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{1,2,3}', '{1.1, 1.2, 1.3}', '{"one", "two", "three"}'),
+ ('{3,1,2}', '{1.3, 1.1, 1.2}', '{"three", "one", "two"}');
+
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (1, 1.2, 1.3, 10),
+ (2, 2.2, 2.3, 20),
+ (3, 3.2, 3.3, 30);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30', 'check replicated data on subscriber');
+
+# Test to reset back to text formatting, and then to binary again
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = false);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (4, 4.2, 4.3, 40);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30
+4|4.2|4.3|40', 'check replicated data on subscriber');
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = true);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{2,3,1}', '{1.2, 1.3, 1.1}', '{"two", "three", "one"}');
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM test_arrays ORDER BY a");
+
+is($result, '{1,2,3}|{1.1,1.2,1.3}|{one,two,three}
+{2,3,1}|{1.2,1.3,1.1}|{two,three,one}
+{3,1,2}|{1.3,1.1,1.2}|{three,one,two}', 'check replicated data on subscriber');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.21.1 (Apple Git-122.3)
On Tue, 7 Jul 2020 at 10:01, Daniel Gustafsson <daniel@yesql.se> wrote:
On 7 Jul 2020, at 02:16, Dave Cramer <davecramer@gmail.com> wrote:
OK, rebased it down to 2 patches, attached.
I took a look at this patchset today. The feature clearly seems like
something
which we'd benefit from having, especially if it allows for the kind of
extensions that were discussed at the beginning of this thread. In
general I
think it's in pretty good shape, there are however a few comments:The patch lacks any kind of test, which I think is required for it to be
considered for committing. It also doesn't update the \dRs view in psql to
include the subbinary column which IMO it should. I took the liberty to
write
this as well as tests as I was playing with the patch, the attached 0003
contains this, while 0001 and 0002 are your patches included to ensure the
CFBot can do it's thing. This was kind of thrown together to have
something
while testing, so it definately need a once-over or two.
I have put all your requests other than the indentation as that can be
dealt with by pg_indent into another patch which I reordered ahead of yours
This should make it easier to see that all of your issues have been
addressed.
I did not do the macro for updated, inserted, deleted, will give that a go
tomorrow.
The comment here implies that unchanged is the default value for format, but isn't this actually setting it to text formatted value? + /* default is unchanged */ + tuple->format = palloc(natts * sizeof(char)); + memset(tuple->format, 't', natts * sizeof(char)); Also, since the values member isn't memset() with a default, this seems a bit misleading at best no?For the rest of the backend we aren't including the defname in the errmsg like what is done here. Maybe we should, but I think that should be done consistently if so, and we should stick to just "conflicting or redundant options" for now. At the very least, this needs a comma between "options" and the defname and ideally the defname wrapped in \". - errmsg("conflicting or redundant options"))); + errmsg("conflicting or redundant options %s already provided", defel->defname)));
I added these since this will now be used outside of logical replication
and getting reasonable error messages when setting up
replication is useful. I added the \" and ,
These types of constructs are IMHO quite hard to read: + if( + #ifdef WORDS_BIGENDIAN + true + #else + false + #endif + != bigendian) How about spelling out the statement completely for both cases, or perhaps encapsulating the logic in a macro? Something like the below perhaps? + #ifdef WORDS_BIGENDIAN + if (bigendian != true) + #else + if (bigendian != false) + #endifThis change needs to be removed before a commit, just highlighting that
here to
avoid it going unnoticed.
-/* #define WAL_DEBUG */
+#define WAL_DEBUGDone
Reading this I'm left wondering if we shoulnd't introduce macros for the
kinds,
since we now compare with 'u', 't' etc in quite a few places and add
comments
explaining the types everywhere. A descriptive name would make it easier
to
grep for all occurrences, and avoid the need for the comment lines. Thats
not
necesarily for this patch though, but an observation from reading it.
I'll take a look at adding macros tomorrow.
I've taken care of much of this below
Show quoted text
I found a few smaller nitpicks as well, some of these might go away by a
pg_indent run but I've included them all here regardless:This change, and the subsequent whitespace removal later in the same
function,
seems a bit pointless:
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-Braces should go on the next line:
+ if (options->proto.logical.binary) {This should be a C /* ... */ comment, or perhaps just removed since the
code
is quite self explanatory:
+ // default to false
+ *binary_basetypes = false;Indentation here: - errmsg("conflicting or redundant options"))); + errmsg("conflicting or redundant options %s already provided", defel->defname)));..as well as here (there are a few like this one): + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("incompatible datum size")));Capitalization of "after" to make it a proper sentence:
+ * after we know that the subscriber is requesting binary check to make
sureExcessive whitespace and indentation in a few places, and not enough in some: + if (binary_given) + { + values[Anum_pg_subscription_subbinary - 1] = ... + if ( *binary_basetypes == true ) ... + if (sizeof(int) != int_size) ... + if( float4_byval != ... + if (sizeof(long) != long_size) + ereport(ERROR, ... + if (tupleData->format[remoteattnum] =='u') ... + bool binary_basetypes;That's all for now.
cheers ./daniel
Attachments:
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patchDownload
From b109ee7d04483b053676755223ba0ccb1b53c511 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:50:21 -0400
Subject: [PATCH 1/4] Add binary protocol for publications and subscriptions
for base types add binary column to pg_subscription support create and alter
subcription with binary option
check that the subscriber is compatible with the
publisher
Removed the array representing binary types in favour of
an array representing the format of each type and use existing codes 't' for
text, 'b' for binary and 'u' for unchanged.
Dynamically allocate the arrays to avoid allocating max columns sized arrays.
Fixed numerous issues with checking for validity
---
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 ++-
.../libpqwalreceiver/libpqwalreceiver.c | 46 ++++
src/backend/replication/logical/proto.c | 122 ++++++----
src/backend/replication/logical/worker.c | 128 +++++++---
src/backend/replication/pgoutput/pgoutput.c | 225 +++++++++++++++++-
src/include/catalog/pg_subscription.h | 4 +
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 20 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
12 files changed, 490 insertions(+), 101 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5314e9348f..9a853fe48d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..f2590ff565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,53 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary) {
+ appendStringInfo(&cmd, ", binary 'true'");
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -161,16 +163,13 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
return relid;
}
@@ -179,7 +178,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +195,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +236,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +258,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +278,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +426,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +442,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +478,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
+
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
{
- int len;
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
+ case 'b': /* binary formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ value->data = palloc(len + 1);
+ /* and data */
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..806e0a8bed 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -396,11 +417,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +465,42 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +666,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +693,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +785,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +820,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +831,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +886,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +929,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +957,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1153,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
+ slot_modify_data(remoteslot_part, localslot,
part_entry,
- newtup->values, newtup->changed);
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2163,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..65c2e5d658 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,30 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
+
+ // default to false
+ *binary_basetypes = false;
+
foreach(lc, options)
{
@@ -138,7 +158,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +179,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,9 +188,197 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
@@ -202,7 +410,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +606,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +621,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +645,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +665,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 8f3ec6bde1..ec4fa01f30 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,10 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,16 +89,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary_basetypes);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..1ee316f6d7 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 83672308a8c37e311c34f2c2f62d61c6b07aadd0 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:53:33 -0400
Subject: [PATCH 2/4] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 11 +++++++++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 003d278370..27b2de6fc7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7503,6 +7503,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..ab712910b2 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..770835edde 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,17 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0003-Actually-set-the-default-to-unchanged-as-per-the-com.patchapplication/octet-stream; name=0003-Actually-set-the-default-to-unchanged-as-per-the-com.patchDownload
From d5fd0e4aab34ddb0d46d0a22ac2780201986d490 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 7 Jul 2020 14:12:20 -0400
Subject: [PATCH 3/4] Actually set the default to unchanged as per the comment
Make more readable and fix up whitespace Fix up error messages
---
.../libpqwalreceiver/libpqwalreceiver.c | 3 +-
src/backend/replication/logical/proto.c | 5 +-
src/backend/replication/pgoutput/pgoutput.c | 54 +++++++++----------
src/include/pg_config_manual.h | 2 +-
4 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 08313fa2a5..7399eec852 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary) {
+ if (options->proto.logical.binary)
+ {
appendStringInfo(&cmd, ", binary 'true'");
appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 73148f39f3..a98d843174 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -163,13 +163,16 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
+ /* read the relation id */
relid = pq_getmsgint(in, 4);
+
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
+
return relid;
}
@@ -525,7 +528,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* default is unchanged */
tuple->format = palloc(natts * sizeof(char));
- memset(tuple->format, 't', natts * sizeof(char));
+ memset(tuple->format, 'u', natts * sizeof(char));
/* Read the data */
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 65c2e5d658..c1c0fc8f8e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -140,7 +140,6 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool float8_byval;
bool integer_datetimes;
- // default to false
*binary_basetypes = false;
@@ -158,7 +157,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -179,7 +178,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -194,7 +193,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (binary_option_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
binary_option_given = true;
if (!parse_bool(strVal(defel->arg), &parsed))
@@ -212,7 +211,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_datum_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_datum_given = true;
if (!scanint8(strVal(defel->arg), true, &datum_size))
@@ -226,7 +225,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_int_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_int_given = true;
if (!scanint8(strVal(defel->arg), true, &int_size))
@@ -240,7 +239,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_long_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_long_given = true;
if (!scanint8(strVal(defel->arg), true, &long_size))
@@ -254,7 +253,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (big_endian_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
big_endian_given = true;
if (!parse_bool(strVal(defel->arg), &bigendian))
@@ -268,7 +267,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float4_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
float4_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float4_byval))
@@ -282,7 +281,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (float8_byval_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
float8_byval_given = true;
if (!parse_bool(strVal(defel->arg), &float8_byval))
@@ -296,7 +295,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (integer_datetimes_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
integer_datetimes_given = true;
if (!parse_bool(strVal(defel->arg), &integer_datetimes))
@@ -328,52 +327,49 @@ parse_output_parameters(List *options, uint32 *protocol_version,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible long size")));
- if(
+
#ifdef WORDS_BIGENDIAN
- true
+ if (bigendian!=true)
#else
- false
+ if (bigendian!=false)
+
#endif
- != bigendian)
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible endianness")));
- if( float4_byval !=
#if PG_VERSION_NUM >= 130000
- true
+ if(float4_byval!=true)
#else
#ifdef USE_FLOAT4_BYVAL
- true
+ if(float4_byval!=true)
#else
- false
+ if(float4_byval!=false)
#endif
#endif
- )
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible float4_byval")));
- if( float8_byval !=
+
+
#ifdef USE_FLOAT8_BYVAL
- true
+ if(float8_byval!=true)
#else
- false
+ if(float8_byval!=false)
#endif
- )
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible float8_byval")));
- if ( integer_datetimes !=
+
#if PG_VERSION_NUM >= 100000
- true
+ if (integer_datetimes!=true)
#else
#ifdef USE_INTEGER_DATETIMES
- true
+ if (integer_datetimes!=true)
#else
- false
+ if (integer_datetimes!=false)
#endif
#endif
- )
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible integer_datetimes")));
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index ec4fa01f30..8f3ec6bde1 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-#define WAL_DEBUG
+/* #define WAL_DEBUG */
/*
* Enable tracing of resource consumption during sort operations;
--
2.20.1 (Apple Git-117)
0004-Add-psql-support-and-tests.patchapplication/octet-stream; name=0004-Add-psql-support-and-tests.patchDownload
From 41e1cc2eaf1294de0981379b879489b76940e8b2 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 7 Jul 2020 11:12:11 -0400
Subject: [PATCH 4/4] Add psql support and tests
---
src/bin/psql/describe.c | 8 +++-
src/test/regress/expected/subscription.out | 56 ++++++++++++++--------
src/test/regress/sql/subscription.sql | 12 +++++
3 files changed, 55 insertions(+), 21 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index cd39b913cd..485c3c6e7c 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5963,7 +5963,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5987,6 +5987,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Enabled"),
gettext_noop("Publication"));
+ /* Binary mode is only supported in v14 and higher */
+ if (pset.sversion >= 140000)
+ appendPQExpBuffer(&buf,
+ ", subbinary AS \"%s\"\n",
+ gettext_noop("Binary"));
+
if (verbose)
{
appendPQExpBuffer(&buf,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index e7add9d2b8..af6ed982ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -91,27 +91,27 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | t | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | t | {testpub2,testpub3} | f
(1 row)
ALTER SUBSCRIPTION regress_testsub DISABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f
(1 row)
COMMIT;
@@ -126,10 +126,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -155,6 +155,22 @@ DROP SUBSCRIPTION IF EXISTS regress_testsub;
NOTICE: subscription "regress_testsub" does not exist, skipping
DROP SUBSCRIPTION regress_testsub; -- fail
ERROR: subscription "regress_testsub" does not exist
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+ERROR: binary requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 9e234ab8b3..835bd05721 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -117,6 +117,18 @@ COMMIT;
DROP SUBSCRIPTION IF EXISTS regress_testsub;
DROP SUBSCRIPTION regress_testsub; -- fail
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
--
2.20.1 (Apple Git-117)
On 7 Jul 2020, at 22:53, Dave Cramer <davecramer@gmail.com> wrote:
I have put all your requests other than the indentation as that can be dealt with by pg_indent into another patch which I reordered ahead of yours
This should make it easier to see that all of your issues have been addressed.
Thanks for the update! Do note that my patch included a new file which is
missing from this patchset:
src/test/subscription/t/014_binary.pl
This is, IMO, the most interesting test of this feature so it would be good to
be included. It's a basic test and can no doubt be extended to be even more
relevant, but it's a start.
I did not do the macro for updated, inserted, deleted, will give that a go tomorrow.
This might not be a blocker, but personally I think it would make the code more
readable. Anyone else have an opinion on this?
I added these since this will now be used outside of logical replication and getting reasonable error messages when setting up
replication is useful. I added the \" and ,
I think the "lack of detail" in the existing error messages is intentional to
make translation easier, but I might be wrong here.
Reading through the new patch, and running the tests, I'm marking this as Ready
for Committer. It does need some cosmetic TLC, quite possibly just from
pg_indent but I didn't validate if it will take care of everything, and comment
touchups (there is still a TODO comment around wording that needs to be
resolved). However, I think it's in good enough shape for consideration at
this point.
cheers ./daniel
Daniel Gustafsson <daniel@yesql.se> writes:
Thanks for the update! Do note that my patch included a new file which is
missing from this patchset:
src/test/subscription/t/014_binary.pl
This is, IMO, the most interesting test of this feature so it would be good to
be included. It's a basic test and can no doubt be extended to be even more
relevant, but it's a start.
I was about to complain that the latest patchset includes no meaningful
test cases, but I assume that this missing file contains those.
I did not do the macro for updated, inserted, deleted, will give that a go tomorrow.
This might not be a blocker, but personally I think it would make the
code more readable. Anyone else have an opinion on this?
+1 for using macros.
Reading through the new patch, and running the tests, I'm marking this as Ready
for Committer. It does need some cosmetic TLC, quite possibly just from
pg_indent but I didn't validate if it will take care of everything, and comment
touchups (there is still a TODO comment around wording that needs to be
resolved). However, I think it's in good enough shape for consideration at
this point.
I took a quick look through the patch and had some concerns:
* Please strip out the PG_VERSION_NUM and USE_INTEGER_DATETIMES checks.
Those are quite dead so far as a patch for HEAD is concerned --- in fact,
since USE_INTEGER_DATETIMES hasn't even been defined since v10 or so,
the patch is actively doing the wrong thing there. Not that it matters.
This code will never appear in any branch where float timestamps could
be a thing.
* I doubt that the checks on USE_FLOAT4/8_BYVAL, sizeof(int), endiannness,
etc, make any sense either. Those surely do not affect the on-the-wire
representation of values --- or if they do, we've blown it somewhere else.
I'd just take out all those checks and assume that the binary
representation is sufficiently portable. (If it's not, it's more or less
the user's problem, just as in binary COPY.)
* Please also remove debugging hacks such as enabling WAL_DEBUG.
* It'd likely be wise for the documentation to point out that binary
mode only works if all types to be transferred have send/receive
functions.
BTW, while it's not the job of this patch to fix it, I find it quite
distressing that we're apparently repeating the lookups of the type
I/O functions for every row transferred.
I'll set this back to WoA, but I concur with Daniel's opinion that
it doesn't seem that far from committability.
regards, tom lane
On Fri, 10 Jul 2020 at 14:21, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Daniel Gustafsson <daniel@yesql.se> writes:
Thanks for the update! Do note that my patch included a new file which
is
missing from this patchset:
src/test/subscription/t/014_binary.pl
This is, IMO, the most interesting test of this feature so it would begood to
be included. It's a basic test and can no doubt be extended to be even
more
relevant, but it's a start.
I was about to complain that the latest patchset includes no meaningful
test cases, but I assume that this missing file contains those.I did not do the macro for updated, inserted, deleted, will give that a
go tomorrow.
This might not be a blocker, but personally I think it would make the
code more readable. Anyone else have an opinion on this?+1 for using macros.
Got it, will add.
Reading through the new patch, and running the tests, I'm marking this
as Ready
for Committer. It does need some cosmetic TLC, quite possibly just from
pg_indent but I didn't validate if it will take care of everything, andcomment
touchups (there is still a TODO comment around wording that needs to be
resolved). However, I think it's in good enough shape for considerationat
this point.
I took a quick look through the patch and had some concerns:
* Please strip out the PG_VERSION_NUM and USE_INTEGER_DATETIMES checks.
Those are quite dead so far as a patch for HEAD is concerned --- in fact,
since USE_INTEGER_DATETIMES hasn't even been defined since v10 or so,
the patch is actively doing the wrong thing there. Not that it matters.
This code will never appear in any branch where float timestamps could
be a thing.* I doubt that the checks on USE_FLOAT4/8_BYVAL, sizeof(int), endiannness,
etc, make any sense either. Those surely do not affect the on-the-wire
representation of values --- or if they do, we've blown it somewhere else.
I'd just take out all those checks and assume that the binary
representation is sufficiently portable. (If it's not, it's more or less
the user's problem, just as in binary COPY.)
So is there any point in having them as options then ?
* Please also remove debugging hacks such as enabling WAL_DEBUG.
* It'd likely be wise for the documentation to point out that binary
mode only works if all types to be transferred have send/receive
functions.
will do
BTW, while it's not the job of this patch to fix it, I find it quite
distressing that we're apparently repeating the lookups of the type
I/O functions for every row transferred.I'll set this back to WoA, but I concur with Daniel's opinion that
it doesn't seem that far from committability.
Thanks for looking at this
Dave Cramer
Hi,
On 11/07/2020 14:14, Dave Cramer wrote:
On Fri, 10 Jul 2020 at 14:21, Tom Lane <tgl@sss.pgh.pa.us
<mailto:tgl@sss.pgh.pa.us>> wrote:Reading through the new patch, and running the tests, I'm marking
this as Ready
for Committer. It does need some cosmetic TLC, quite possibly
just from
pg_indent but I didn't validate if it will take care of
everything, and comment
touchups (there is still a TODO comment around wording that needs
to be
resolved). However, I think it's in good enough shape for
consideration at
this point.
I took a quick look through the patch and had some concerns:
* Please strip out the PG_VERSION_NUM and USE_INTEGER_DATETIMES checks.
Those are quite dead so far as a patch for HEAD is concerned --- in
fact,
since USE_INTEGER_DATETIMES hasn't even been defined since v10 or so,
the patch is actively doing the wrong thing there. Not that it matters.
This code will never appear in any branch where float timestamps could
be a thing.* I doubt that the checks on USE_FLOAT4/8_BYVAL, sizeof(int),
endiannness,
etc, make any sense either. Those surely do not affect the on-the-wire
representation of values --- or if they do, we've blown it somewhere
else.
I'd just take out all those checks and assume that the binary
representation is sufficiently portable. (If it's not, it's more or
less
the user's problem, just as in binary COPY.)So is there any point in having them as options then ?
I am guessing this is copied from pglogical, right? We have them there
because it can optionally send data in the on-disk format (not the
network binary format) and there this matters, but for network binary
format they do not matter as Tom says.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Petr Jelinek <petr@2ndquadrant.com> writes:
On 11/07/2020 14:14, Dave Cramer wrote:
So is there any point in having them as options then ?
I am guessing this is copied from pglogical, right? We have them there
because it can optionally send data in the on-disk format (not the
network binary format) and there this matters, but for network binary
format they do not matter as Tom says.
Ah, I wondered why that was there at all. Yes, you should just delete
all that logic --- it's irrelevant as long as we use the send/recv
functions.
regards, tom lane
On Sat, 11 Jul 2020 at 10:20, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Petr Jelinek <petr@2ndquadrant.com> writes:
On 11/07/2020 14:14, Dave Cramer wrote:
So is there any point in having them as options then ?
I am guessing this is copied from pglogical, right? We have them there
because it can optionally send data in the on-disk format (not the
network binary format) and there this matters, but for network binary
format they do not matter as Tom says.Ah, I wondered why that was there at all. Yes, you should just delete
all that logic --- it's irrelevant as long as we use the send/recv
functions.regards, tom lane
Ok,
removed all the unnecessary options.
Added the test case that Daniel had created.
Added a note to the docs.
Note WAL_DEBUG is removed in patch 3. I could rebase that into patch 1 if
required.
Thanks,
Dave
Attachments:
0003-Actually-set-the-default-to-unchanged-as-per-the-com.patchapplication/octet-stream; name=0003-Actually-set-the-default-to-unchanged-as-per-the-com.patchDownload
From 05789a219a87dccfd9a84c24332d5c655a2f524c Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 7 Jul 2020 14:12:20 -0400
Subject: [PATCH 3/4] Actually set the default to unchanged as per the comment
Make more readable and fix up whitespace Fix up error messages
remove useless options. Add macros for unchanged, binary and text
---
.../libpqwalreceiver/libpqwalreceiver.c | 44 +------
src/backend/replication/logical/proto.c | 25 ++--
src/backend/replication/logical/worker.c | 8 +-
src/backend/replication/pgoutput/pgoutput.c | 117 ++----------------
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 4 +
6 files changed, 35 insertions(+), 165 deletions(-)
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 08313fa2a5..a571aada71 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,52 +423,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
- if (options->proto.logical.binary) {
+ if (options->proto.logical.binary)
+ {
appendStringInfo(&cmd, ", binary 'true'");
appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
- appendStringInfo(&cmd, ", bigendian '%d'",
-#ifdef WORDS_BIGENDIAN
- true
-#else
- false
-#endif
- );
- appendStringInfo(&cmd, ", float4_byval '%d'",
-#if PG_VERSION_NUM >= 130000
- true
-#else
-#ifdef USE_FLOAT4_BYVAL
- true
-#else
- false
-#endif
-#endif
- );
- appendStringInfo(&cmd, ", float8_byval '%d'",
-#ifdef USE_FLOAT8_BYVAL
- true
-#else
- false
-#endif
- );
- appendStringInfo(&cmd, ", integer_datetimes '%d'",
-
-/* integer date times are always enabled in version 10 and up */
-
-#if PG_VERSION_NUM >= 100000
- true
-#else
-
-#ifdef USE_INTEGER_DATETIMES
- true
-#else
- false
-#endif
-#endif
-
- );
}
appendStringInfoChar(&cmd, ')');
}
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 73148f39f3..521f0139a2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -163,13 +163,16 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
+ /* read the relation id */
relid = pq_getmsgint(in, 4);
+
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
+
return relid;
}
@@ -469,7 +472,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
}
else if (att->attlen == -1 && VARATT_IS_EXTERNAL_ONDISK(values[i]))
{
- pq_sendbyte(out, 'u'); /* unchanged toast column */
+ pq_sendbyte(out, LOGICALREP_UNCHANGED);
continue;
}
@@ -486,7 +489,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
{
bytea *outputbytes;
int len;
- pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+ pq_sendbyte(out, LOGICALREP_BINARY);
outputbytes = OidSendFunctionCall(typclass->typsend,
values[i]);
@@ -498,7 +501,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
}
else
{
- pq_sendbyte(out, 't'); /* 'text' data follows */
+ pq_sendbyte(out, LOGICALREP_TEXT);
outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
pfree(outputstr);
@@ -525,7 +528,7 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* default is unchanged */
tuple->format = palloc(natts * sizeof(char));
- memset(tuple->format, 't', natts * sizeof(char));
+ memset(tuple->format, LOGICALREP_UNCHANGED, natts * sizeof(char));
/* Read the data */
@@ -540,21 +543,21 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
case 'n': /* null */
{
tuple->values[i] = (StringInfoData *)NULL;
- tuple->format[i] = 't';
+ tuple->format[i] = LOGICALREP_TEXT;
break;
}
- case 'u': /* unchanged column */
+ case LOGICALREP_UNCHANGED:
{
/* we don't receive the value of an unchanged column */
tuple->values[i] = (StringInfoData *)NULL;
- tuple->format[i] = 'u'; /* be explicit */
+ tuple->format[i] = LOGICALREP_UNCHANGED; /* be explicit */
break;
}
- case 'b': /* binary formatted value */
+ case LOGICALREP_BINARY:
{
int len;
StringInfoData *value = palloc(sizeof(StringInfoData));
- tuple->format[i] = 'b';
+ tuple->format[i] = LOGICALREP_BINARY;
len = pq_getmsgint(in, 4); /* read length */
@@ -570,11 +573,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
tuple->values[i] = value;
break;
}
- case 't': /* text formatted value */
+ case LOGICALREP_TEXT:
{
int len;
StringInfoData *value = palloc(sizeof(StringInfoData));
- tuple->format[i] = 't';
+ tuple->format[i] = LOGICALREP_TEXT;
len = pq_getmsgint(in, 4); /* read length */
/* and data */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 806e0a8bed..40952edf18 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -356,7 +356,7 @@ slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->format[remoteattnum] == 'b')
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
{
Oid typreceive;
Oid typioparam;
@@ -465,7 +465,7 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (tupleData->format[remoteattnum] =='u')
+ if (tupleData->format[remoteattnum] == LOGICALREP_UNCHANGED)
continue;
if (tupleData->values[remoteattnum] != NULL)
@@ -474,7 +474,7 @@ slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- if (tupleData->format[remoteattnum] == 'b')
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
{
Oid typreceive;
Oid typioparam;
@@ -820,7 +820,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.format[i] != 'u')
+ if (newtup.format[i] != LOGICALREP_UNCHANGED)
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 65c2e5d658..837f438e53 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -140,7 +140,6 @@ parse_output_parameters(List *options, uint32 *protocol_version,
bool float8_byval;
bool integer_datetimes;
- // default to false
*binary_basetypes = false;
@@ -158,7 +157,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -179,7 +178,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -194,7 +193,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (binary_option_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
binary_option_given = true;
if (!parse_bool(strVal(defel->arg), &parsed))
@@ -212,7 +211,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_datum_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_datum_given = true;
if (!scanint8(strVal(defel->arg), true, &datum_size))
@@ -226,7 +225,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_int_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_int_given = true;
if (!scanint8(strVal(defel->arg), true, &int_size))
@@ -240,7 +239,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (sizeof_long_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
sizeof_long_given = true;
if (!scanint8(strVal(defel->arg), true, &long_size))
@@ -248,62 +247,6 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid sizeof_long option")));
}
- else if (strcmp(defel->defname, "bigendian") == 0)
- {
-
- if (big_endian_given)
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
- big_endian_given = true;
-
- if (!parse_bool(strVal(defel->arg), &bigendian))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid bigendian option")));
- }
- else if (strcmp(defel->defname, "float4_byval") == 0)
- {
-
- if (float4_byval_given)
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
- float4_byval_given = true;
-
- if (!parse_bool(strVal(defel->arg), &float4_byval))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid float4_byval option")));
- }
- else if (strcmp(defel->defname, "float8_byval") == 0)
- {
-
- if (float8_byval_given)
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
- float8_byval_given = true;
-
- if (!parse_bool(strVal(defel->arg), &float8_byval))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid float8_byval option")));
- }
- else if (strcmp(defel->defname, "integer_datetimes") == 0)
- {
-
- if (integer_datetimes_given)
- ereport(ERROR,
- (errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options %s already provided", defel->defname)));
- integer_datetimes_given = true;
-
- if (!parse_bool(strVal(defel->arg), &integer_datetimes))
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid integer_date_times option")));
- }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -328,56 +271,16 @@ parse_output_parameters(List *options, uint32 *protocol_version,
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("incompatible long size")));
- if(
+
#ifdef WORDS_BIGENDIAN
- true
-#else
- false
-#endif
- != bigendian)
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("incompatible endianness")));
- if( float4_byval !=
-#if PG_VERSION_NUM >= 130000
- true
-#else
-#ifdef USE_FLOAT4_BYVAL
- true
+ if (bigendian!=true)
#else
- false
-#endif
-#endif
- )
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("incompatible float4_byval")));
- if( float8_byval !=
-#ifdef USE_FLOAT8_BYVAL
- true
-#else
- false
-#endif
- )
- ereport(ERROR,
- (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("incompatible float8_byval")));
+ if (bigendian!=false)
- if ( integer_datetimes !=
-#if PG_VERSION_NUM >= 100000
- true
-#else
-#ifdef USE_INTEGER_DATETIMES
- true
-#else
- false
#endif
-#endif
- )
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("incompatible integer_datetimes")));
-
+ errmsg("incompatible endianness")));
}
}
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index ec4fa01f30..8f3ec6bde1 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-#define WAL_DEBUG
+/* #define WAL_DEBUG */
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index b209af4cf2..06b56b253f 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -27,6 +27,10 @@
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_UNCHANGED 'u'
+#define LOGICALREP_BINARY 'b'
+#define LOGICALREP_TEXT 't'
+
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
--
2.20.1 (Apple Git-117)
0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchapplication/octet-stream; name=0002-document-new-binary-option-for-CREATE-SUBSCRIPTION.patchDownload
From 8855262313b6d5ecc282a21644dc0d1884fca39c Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:53:33 -0400
Subject: [PATCH 2/4] document new binary option for CREATE SUBSCRIPTION
document addition of binary column to pg_subscription
Add note to documentation that transferring a type using binary protocol requires that the type have a send and recieve function
---
doc/src/sgml/catalogs.sgml | 7 +++++++
doc/src/sgml/ref/alter_subscription.sgml | 4 ++--
doc/src/sgml/ref/create_subscription.sgml | 16 ++++++++++++++++
3 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 003d278370..27b2de6fc7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7503,6 +7503,13 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry><structfield>subbinary</structfield></entry>
+ <entry><type>bool</type></entry>
+ <entry></entry>
+ <entry>If true, the subscription will request that the publisher send base types in binary format.</entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..ab712910b2 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..35dfaf017f 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,22 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher send
+ the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+
+ <para>
+ Note: Only types that have send and receive functions will be transferred
+ in binary
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
--
2.20.1 (Apple Git-117)
0004-Add-psql-support-and-tests.patchapplication/octet-stream; name=0004-Add-psql-support-and-tests.patchDownload
From b38d729ab32f733c494bf03c0b4760b8b2dc9b79 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Tue, 7 Jul 2020 11:12:11 -0400
Subject: [PATCH 4/4] Add psql support and tests
---
src/bin/psql/describe.c | 8 +++-
src/test/regress/expected/subscription.out | 56 ++++++++++++++--------
src/test/regress/sql/subscription.sql | 12 +++++
3 files changed, 55 insertions(+), 21 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index cd39b913cd..485c3c6e7c 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5963,7 +5963,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5987,6 +5987,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Enabled"),
gettext_noop("Publication"));
+ /* Binary mode is only supported in v14 and higher */
+ if (pset.sversion >= 140000)
+ appendPQExpBuffer(&buf,
+ ", subbinary AS \"%s\"\n",
+ gettext_noop("Binary"));
+
if (verbose)
{
appendPQExpBuffer(&buf,
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index e7add9d2b8..af6ed982ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -91,27 +91,27 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | t | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | t | {testpub2,testpub3} | f
(1 row)
ALTER SUBSCRIPTION regress_testsub DISABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f
(1 row)
COMMIT;
@@ -126,10 +126,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -155,6 +155,22 @@ DROP SUBSCRIPTION IF EXISTS regress_testsub;
NOTICE: subscription "regress_testsub" does not exist, skipping
DROP SUBSCRIPTION regress_testsub; -- fail
ERROR: subscription "regress_testsub" does not exist
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+ERROR: binary requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 9e234ab8b3..835bd05721 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -117,6 +117,18 @@ COMMIT;
DROP SUBSCRIPTION IF EXISTS regress_testsub;
DROP SUBSCRIPTION regress_testsub; -- fail
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
--
2.20.1 (Apple Git-117)
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patchDownload
From b109ee7d04483b053676755223ba0ccb1b53c511 Mon Sep 17 00:00:00 2001
From: Dave Cramer <davecramer@gmail.com>
Date: Mon, 6 Jul 2020 19:50:21 -0400
Subject: [PATCH 1/4] Add binary protocol for publications and subscriptions
for base types add binary column to pg_subscription support create and alter
subcription with binary option
check that the subscriber is compatible with the
publisher
Removed the array representing binary types in favour of
an array representing the format of each type and use existing codes 't' for
text, 'b' for binary and 'u' for unchanged.
Dynamically allocate the arrays to avoid allocating max columns sized arrays.
Fixed numerous issues with checking for validity
---
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 39 ++-
.../libpqwalreceiver/libpqwalreceiver.c | 46 ++++
src/backend/replication/logical/proto.c | 122 ++++++----
src/backend/replication/logical/worker.c | 128 +++++++---
src/backend/replication/pgoutput/pgoutput.c | 225 +++++++++++++++++-
src/include/catalog/pg_subscription.h | 4 +
src/include/pg_config_manual.h | 2 +-
src/include/replication/logicalproto.h | 20 +-
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
12 files changed, 490 insertions(+), 101 deletions(-)
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 5314e9348f..9a853fe48d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1125,7 +1125,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..f2590ff565 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,12 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+ /* not all versions of pgoutput will understand this option default to false */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +181,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -324,8 +335,10 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool slotname_given;
char originname[NAMEDATALEN];
bool create_slot;
- List *publications;
+ bool binary;
+ bool binary_given;
+ List *publications;
/*
* Parse and check options.
*
@@ -334,7 +347,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +413,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +683,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +714,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +732,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +771,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +808,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..08313fa2a5 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,53 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
+ if (options->proto.logical.binary) {
+ appendStringInfo(&cmd, ", binary 'true'");
+ appendStringInfo(&cmd, ", sizeof_datum '%zu'", sizeof(Datum));
+ appendStringInfo(&cmd, ", sizeof_int '%zu'", sizeof(int));
+ appendStringInfo(&cmd, ", sizeof_long '%zu'", sizeof(long));
+ appendStringInfo(&cmd, ", bigendian '%d'",
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", float4_byval '%d'",
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ );
+ appendStringInfo(&cmd, ", float8_byval '%d'",
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ );
+ appendStringInfo(&cmd, ", integer_datetimes '%d'",
+
+/* integer date times are always enabled in version 10 and up */
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+
+ );
+ }
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..73148f39f3 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -16,9 +16,11 @@
#include "catalog/pg_namespace.h"
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
+#include "replication/logicalrelation.h"
#include "replication/logicalproto.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -161,16 +163,13 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
char action;
LogicalRepRelId relid;
- /* read the relation id */
relid = pq_getmsgint(in, 4);
-
action = pq_getmsgbyte(in);
if (action != 'N')
elog(ERROR, "expected new tuple but got %d",
action);
logicalrep_read_tuple(in, newtup);
-
return relid;
}
@@ -179,7 +178,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +195,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -241,14 +236,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
logicalrep_read_tuple(in, newtup);
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +258,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -288,7 +278,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
logicalrep_read_tuple(in, oldtup);
- return relid;
}
/*
@@ -437,7 +426,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +442,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -488,12 +478,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+ pq_sendbyte(out, 'b'); /* binary send/recv data follows */
+
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, 't'); /* 'text' data follows */
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +521,12 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, 't', natts * sizeof(char));
+
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,51 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
+ {
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 't';
+ break;
+ }
case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
{
- int len;
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *)NULL;
+ tuple->format[i] = 'u'; /* be explicit */
+ break;
+ }
+ case 'b': /* binary formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 'b';
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ value->data = palloc(len + 1);
+ /* and data */
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case 't': /* text formatted value */
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = 't';
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..806e0a8bed 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,40 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -396,11 +417,17 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* Caution: unreplaced pass-by-ref columns in "slot" will point into the
* storage for "srcslot". This is OK for current usage, but someday we may
* need to materialize "slot" at the end to make it independent of "srcslot".
+ *
+ * TODO: figure out the right comment here
+ * Modify slot with user data provided.
+ * This is somewhat similar to heap_modify_tuple but also calls the type
+ * input function on the user data as the input is either text or binary transfer
+ * format
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +465,42 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] =='u')
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == 'b')
+ {
+ Oid typreceive;
+ Oid typioparam;
+
+ int cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +666,12 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +693,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +785,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +820,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != 'u')
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +831,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +886,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +929,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +957,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1153,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
+ slot_modify_data(remoteslot_part, localslot,
part_entry,
- newtup->values, newtup->changed);
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2163,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..65c2e5d658 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,30 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+ bool sizeof_int_given = false;
+ bool sizeof_datum_given = false;
+ bool sizeof_long_given = false;
+ bool big_endian_given = false;
+ bool float4_byval_given = false;
+ bool float8_byval_given = false;
+ bool integer_datetimes_given = false;
+ long datum_size;
+ long int_size;
+ long long_size;
+ bool bigendian;
+ bool float4_byval;
+ bool float8_byval;
+ bool integer_datetimes;
+
+ // default to false
+ *binary_basetypes = false;
+
foreach(lc, options)
{
@@ -138,7 +158,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +179,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,9 +188,197 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0 )
+ {
+ bool parsed;
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+
+ *binary_basetypes = parsed;
+
+
+ }
+ else if (strcmp(defel->defname, "sizeof_datum") == 0)
+ {
+
+ if (sizeof_datum_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_datum_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &datum_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_datum option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_int") == 0)
+ {
+
+ if (sizeof_int_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_int_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &int_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_int option")));
+ }
+ else if (strcmp(defel->defname, "sizeof_long") == 0)
+ {
+
+ if (sizeof_long_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ sizeof_long_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &long_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid sizeof_long option")));
+ }
+ else if (strcmp(defel->defname, "bigendian") == 0)
+ {
+
+ if (big_endian_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ big_endian_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &bigendian))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid bigendian option")));
+ }
+ else if (strcmp(defel->defname, "float4_byval") == 0)
+ {
+
+ if (float4_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float4_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float4_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float4_byval option")));
+ }
+ else if (strcmp(defel->defname, "float8_byval") == 0)
+ {
+
+ if (float8_byval_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ float8_byval_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &float8_byval))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid float8_byval option")));
+ }
+ else if (strcmp(defel->defname, "integer_datetimes") == 0)
+ {
+
+ if (integer_datetimes_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options %s already provided", defel->defname)));
+ integer_datetimes_given = true;
+
+ if (!parse_bool(strVal(defel->arg), &integer_datetimes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid integer_date_times option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
+
+/*
+ * after we know that the subscriber is requesting binary check to make sure
+ * we are compatible with the subscriber.
+ */
+ if ( *binary_basetypes == true )
+ {
+ if (sizeof(Datum) != datum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible datum size")));
+
+ if (sizeof(int) != int_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer size")));
+
+ if (sizeof(long) != long_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible long size")));
+ if(
+#ifdef WORDS_BIGENDIAN
+ true
+#else
+ false
+#endif
+ != bigendian)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible endianness")));
+ if( float4_byval !=
+#if PG_VERSION_NUM >= 130000
+ true
+#else
+#ifdef USE_FLOAT4_BYVAL
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float4_byval")));
+ if( float8_byval !=
+#ifdef USE_FLOAT8_BYVAL
+ true
+#else
+ false
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible float8_byval")));
+
+ if ( integer_datetimes !=
+#if PG_VERSION_NUM >= 100000
+ true
+#else
+#ifdef USE_INTEGER_DATETIMES
+ true
+#else
+ false
+#endif
+#endif
+ )
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("incompatible integer_datetimes")));
+
+ }
}
/*
@@ -202,7 +410,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +606,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +621,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +645,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +665,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..100789a52f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the
+ * output plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pg_config_manual.h b/src/include/pg_config_manual.h
index 8f3ec6bde1..ec4fa01f30 100644
--- a/src/include/pg_config_manual.h
+++ b/src/include/pg_config_manual.h
@@ -336,7 +336,7 @@
* Enable debugging print statements for WAL-related operations; see
* also the wal_debug GUC var.
*/
-/* #define WAL_DEBUG */
+#define WAL_DEBUG
/*
* Enable tracing of resource consumption during sort operations;
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..b209af4cf2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -30,10 +30,12 @@
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,16 +89,16 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary_basetypes);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..1252e9132b 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..1ee316f6d7 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1 (Apple Git-117)
On 13 Jul 2020, at 15:11, Dave Cramer <davecramer@gmail.com> wrote:
I took another look at the updated version today. Since there now were some
unused variables and (I believe) unnecessary checks (int size and endianness
etc) left, I took the liberty to fix those. I also fixed some markup in the
catalog docs, did some minor tidying up and ran pgindent on it.
The attached is a squash of the 4 patches in your email with the above fixes.
I'm again marking it RfC since I believe all concerns raised so far has been
addressed.
Added the test case that Daniel had created.
Nope, still missing AFAICT =) But I've included it in the attached.
cheers ./daniel
Attachments:
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patch; x-unix-mode=0644Download
From 6a961ebaca47eb54bb20f2cdba1811653274d21b Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 14 Jul 2020 13:54:37 +0200
Subject: [PATCH] Add binary protocol for publications and subscriptions
This adds support for transferring base types using binary format between
subscriber and publisher. A new column is added to pg_subscription, subbinary,
in order to track whether binary is enabled or not.
Author: Dave Cramer
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 16 +++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 41 +++++-
.../libpqwalreceiver/libpqwalreceiver.c | 3 +-
src/backend/replication/logical/proto.c | 123 ++++++++++------
src/backend/replication/logical/worker.c | 134 ++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 34 ++++-
src/bin/psql/describe.c | 8 +-
src/include/catalog/pg_subscription.h | 5 +
src/include/replication/logicalproto.h | 30 ++--
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
src/test/regress/expected/subscription.out | 56 +++++---
src/test/regress/sql/subscription.sql | 12 ++
src/test/subscription/t/014_binary.pl | 108 ++++++++++++++
18 files changed, 458 insertions(+), 131 deletions(-)
create mode 100644 src/test/subscription/t/014_binary.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e9cdff4864..fe317ec37f 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7504,6 +7504,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subbinary</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will request that the publisher send base
+ types in binary format.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..98ca11cb1c 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..1d71368254 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,22 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher to
+ send the base types in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+
+ <para>
+ Note: Only types that have send and receive functions will be
+ transferred in binary
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b6d35c2d11..7f02d6dbb2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1122,7 +1122,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..d87de2ac2f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+
+ /*
+ * Not all versions of pgoutput will understand this option, so
+ * default to false.
+ */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +185,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +340,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
char originname[NAMEDATALEN];
bool create_slot;
List *publications;
+ bool binary;
+ bool binary_given;
/*
* Parse and check options.
@@ -334,7 +351,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +417,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +687,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +718,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +736,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +775,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +812,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..7989b58019 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
-
+ if (options->proto.logical.binary)
+ appendStringInfoString(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..b91ca2714d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -17,8 +17,10 @@
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary_basetypes);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
@@ -179,7 +181,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary_basetypes)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +198,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary_basetypes);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -240,15 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
action);
logicalrep_read_tuple(in, newtup);
-
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary_basetypes)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary_basetypes);
}
/*
@@ -272,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -287,8 +279,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
elog(ERROR, "expected action 'O' or 'K', got %c", action);
logicalrep_read_tuple(in, oldtup);
-
- return relid;
}
/*
@@ -437,7 +427,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary_basetypes)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +443,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -479,7 +470,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
}
else if (att->attlen == -1 && VARATT_IS_EXTERNAL_ONDISK(values[i]))
{
- pq_sendbyte(out, 'u'); /* unchanged toast column */
+ pq_sendbyte(out, LOGICALREP_UNCHANGED);
continue;
}
@@ -488,12 +479,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
+ if (binary_basetypes &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+
+ pq_sendbyte(out, LOGICALREP_BINARY);
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, LOGICALREP_TEXT);
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +522,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, LOGICALREP_UNCHANGED, natts * sizeof(char));
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,52 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
- case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
+ {
+ tuple->values[i] = (StringInfoData *) NULL;
+ tuple->format[i] = LOGICALREP_TEXT;
+ break;
+ }
+ case LOGICALREP_UNCHANGED:
+ {
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *) NULL;
+ tuple->format[i] = LOGICALREP_UNCHANGED; /* be explicit */
+ break;
+ }
+ case LOGICALREP_BINARY:
{
int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+
+ tuple->format[i] = LOGICALREP_BINARY;
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ /* and data */
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case LOGICALREP_TEXT:
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = LOGICALREP_TEXT;
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f90a896fc3..844845c473 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,42 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
+ {
+ Oid typreceive;
+ Oid typioparam;
+ int cursor;
+
+ cursor = tupleData->values[remoteattnum]->cursor;
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into
+ * proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -387,9 +410,9 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Replace selected columns with user data provided as C strings.
- * This is somewhat similar to heap_modify_tuple but also calls the type
- * input functions on the user data.
+ * Replace selected columns with user data provided either as C strings or in
+ * binary format. This is somewhat similar to heap_modify_tuple but also calls
+ * the type input functions on the user data.
* "slot" is filled with a copy of the tuple in "srcslot", with
* columns selected by the "replaces" array replaced with data values
* from "values".
@@ -398,9 +421,9 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* need to materialize "slot" at the end to make it independent of "srcslot".
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +461,44 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] == LOGICALREP_UNCHANGED)
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
-
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
+ {
+ Oid typreceive;
+ Oid typioparam;
+ int cursor;
+
+ cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into
+ * proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +664,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +689,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +781,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +816,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != LOGICALREP_UNCHANGED)
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +827,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +882,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +925,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +953,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1149,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
- part_entry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot_part, localslot,
+ part_entry,
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2159,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..e209ad8aa5 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,14 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary_basetypes)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ *binary_basetypes = false;
foreach(lc, options)
{
@@ -138,7 +142,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +163,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,6 +172,19 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0)
+ {
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), binary_basetypes))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -202,7 +219,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary_basetypes);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +415,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +430,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +454,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary_basetypes);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +474,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary_basetypes);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 3b870c3b17..a7fd29085f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5963,7 +5963,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5987,6 +5987,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Enabled"),
gettext_noop("Publication"));
+ /* Binary mode is only supported in v14 and higher */
+ if (pset.sversion >= 140000)
+ appendPQExpBuffer(&buf,
+ ", subbinary AS \"%s\"\n",
+ gettext_noop("Binary"));
+
if (verbose)
{
appendPQExpBuffer(&buf,
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..e8b3712574 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the output
+ * plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,8 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in
+ * binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..3971f007ae 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -27,13 +27,19 @@
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_UNCHANGED 'u'
+#define LOGICALREP_BINARY 'b'
+#define LOGICALREP_TEXT 't'
+
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,17 +93,17 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary_basetypes);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
- bool *has_oldtuple, LogicalRepTupleData *oldtup,
- LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary_basetypes);
+extern void logicalrep_read_update(StringInfo in,
+ bool *has_oldtuple, LogicalRepTupleData *oldtup,
+ LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
- LogicalRepTupleData *oldtup);
+ HeapTuple oldtuple, bool binary_basetypes);
+extern void logicalrep_read_delete(StringInfo in,
+ LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..a2d76689ed 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary_basetypes;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..7cd1458819 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index e7add9d2b8..af6ed982ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -91,27 +91,27 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | t | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | t | {testpub2,testpub3} | f
(1 row)
ALTER SUBSCRIPTION regress_testsub DISABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f
(1 row)
COMMIT;
@@ -126,10 +126,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -155,6 +155,22 @@ DROP SUBSCRIPTION IF EXISTS regress_testsub;
NOTICE: subscription "regress_testsub" does not exist, skipping
DROP SUBSCRIPTION regress_testsub; -- fail
ERROR: subscription "regress_testsub" does not exist
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+ERROR: binary requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 9e234ab8b3..835bd05721 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -117,6 +117,18 @@ COMMIT;
DROP SUBSCRIPTION IF EXISTS regress_testsub;
DROP SUBSCRIPTION regress_testsub; -- fail
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/014_binary.pl b/src/test/subscription/t/014_binary.pl
new file mode 100644
index 0000000000..a252f470ef
--- /dev/null
+++ b/src/test/subscription/t/014_binary.pl
@@ -0,0 +1,108 @@
+# Binary mode logical replication test
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Create and initialize a publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create and initialize subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+my $ddl = qq(
+ CREATE TABLE public.test_numerical (
+ a INTEGER PRIMARY KEY,
+ b NUMERIC,
+ c FLOAT,
+ d BIGINT
+ );
+ CREATE TABLE public.test_arrays (
+ a INTEGER[] PRIMARY KEY,
+ b NUMERIC[],
+ c TEXT[]
+ ););
+
+# Create content on both sides of the replication
+$node_publisher->safe_psql('postgres', $ddl);
+$node_subscriber->safe_psql('postgres', $ddl);
+
+# Configure logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tpub FOR ALL TABLES");
+
+my $publisher_connstring = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tsub CONNECTION '$publisher_connstring' " .
+ "PUBLICATION tpub WITH (slot_name = tpub_slot, binary = true)");
+
+# Ensure nodes are in sync with eachother
+$node_publisher->wait_for_catchup('tsub');
+$node_subscriber->poll_query_until('postgres',
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');")
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Insert some content and make sure it's replicated across
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{1,2,3}', '{1.1, 1.2, 1.3}', '{"one", "two", "three"}'),
+ ('{3,1,2}', '{1.3, 1.1, 1.2}', '{"three", "one", "two"}');
+
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (1, 1.2, 1.3, 10),
+ (2, 2.2, 2.3, 20),
+ (3, 3.2, 3.3, 30);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30', 'check replicated data on subscriber');
+
+# Test to reset back to text formatting, and then to binary again
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = false);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (4, 4.2, 4.3, 40);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30
+4|4.2|4.3|40', 'check replicated data on subscriber');
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = true);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{2,3,1}', '{1.2, 1.3, 1.1}', '{"two", "three", "one"}');
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM test_arrays ORDER BY a");
+
+is($result, '{1,2,3}|{1.1,1.2,1.3}|{one,two,three}
+{2,3,1}|{1.2,1.3,1.1}|{two,three,one}
+{3,1,2}|{1.3,1.1,1.2}|{three,one,two}', 'check replicated data on subscriber');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.21.1 (Apple Git-122.3)
On Tue, 14 Jul 2020 at 09:26, Daniel Gustafsson <daniel@yesql.se> wrote:
On 13 Jul 2020, at 15:11, Dave Cramer <davecramer@gmail.com> wrote:
I took another look at the updated version today. Since there now were
some
unused variables and (I believe) unnecessary checks (int size and
endianness
etc) left, I took the liberty to fix those. I also fixed some markup in
the
catalog docs, did some minor tidying up and ran pgindent on it.The attached is a squash of the 4 patches in your email with the above
fixes.
I'm again marking it RfC since I believe all concerns raised so far has
been
addressed.Added the test case that Daniel had created.
Nope, still missing AFAICT =) But I've included it in the attached.
Thanks!
Dave
Show quoted text
So I started looking through this seriously, and my first question
is why do the docs and code keep saying that "base types" are sent
in binary? Why not just "data"? Are there any cases where we
don't use binary format, if the subscription requests it?
If there's not a concrete reason to use that terminology,
I'd rather flush it, because it seems confusing.
regards, tom lane
On Tue, 14 Jul 2020 at 12:59, Tom Lane <tgl@sss.pgh.pa.us> wrote:
So I started looking through this seriously, and my first question
is why do the docs and code keep saying that "base types" are sent
in binary? Why not just "data"? Are there any cases where we
don't use binary format, if the subscription requests it?If there's not a concrete reason to use that terminology,
I'd rather flush it, because it seems confusing.
Well for some reason I thought there were some types that did not have send
and receive functions.
I've changed the docs to say data and the flag from binary_basetypes to
just binary
See attached.
Thanks,
Dave
Show quoted text
Attachments:
0001-Add-binary-protocol-for-publications-and-subscriptio.patchapplication/octet-stream; name=0001-Add-binary-protocol-for-publications-and-subscriptio.patchDownload
From 6a961ebaca47eb54bb20f2cdba1811653274d21b Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <daniel@yesql.se>
Date: Tue, 14 Jul 2020 13:54:37 +0200
Subject: [PATCH] Add binary protocol for publications and subscriptions
This adds support for transferring data using binary format between
subscriber and publisher. A new column is added to pg_subscription, subbinary,
in order to track whether binary is enabled or not.
Author: Dave Cramer
---
doc/src/sgml/catalogs.sgml | 10 ++
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 16 +++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/subscriptioncmds.c | 41 +++++-
.../libpqwalreceiver/libpqwalreceiver.c | 3 +-
src/backend/replication/logical/proto.c | 123 ++++++++++------
src/backend/replication/logical/worker.c | 134 ++++++++++++------
src/backend/replication/pgoutput/pgoutput.c | 34 ++++-
src/bin/psql/describe.c | 8 +-
src/include/catalog/pg_subscription.h | 5 +
src/include/replication/logicalproto.h | 30 ++--
src/include/replication/pgoutput.h | 1 +
src/include/replication/walreceiver.h | 1 +
src/test/regress/expected/subscription.out | 56 +++++---
src/test/regress/sql/subscription.sql | 12 ++
src/test/subscription/t/014_binary.pl | 108 ++++++++++++++
18 files changed, 458 insertions(+), 131 deletions(-)
create mode 100644 src/test/subscription/t/014_binary.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index e9cdff4864..fe317ec37f 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -7504,6 +7504,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
</para></entry>
</row>
+ <row>
+ <entry role="catalog_table_entry"><para role="column_definition">
+ <structfield>subbinary</structfield> <type>bool</type>
+ </para>
+ <para>
+ If true, the subscription will request that the publisher send data
+ in binary format.
+ </para></entry>
+ </row>
+
<row>
<entry role="catalog_table_entry"><para role="column_definition">
<structfield>subconninfo</structfield> <type>text</type>
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index c24ace14d1..98ca11cb1c 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>binary</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 5bbc165f70..1d71368254 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -128,6 +128,22 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>binary</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether the subscription will request the publisher to
+ send the data in binary or not. The default
+ is <literal>false</literal>.
+ </para>
+
+ <para>
+ Note: Only types that have send and receive functions will be
+ transferred in binary
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>slot_name</literal> (<type>string</type>)</term>
<listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731115..e6afb3203e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->binary = subform->subbinary;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b6d35c2d11..7f02d6dbb2 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1122,7 +1122,7 @@ REVOKE ALL ON pg_replication_origin_status FROM public;
-- All columns of pg_subscription except subconninfo are readable.
REVOKE ALL ON pg_subscription FROM public;
-GRANT SELECT (subdbid, subname, subowner, subenabled, subslotname, subpublications)
+GRANT SELECT (subdbid, subname, subowner, subenabled, subbinary, subslotname, subpublications)
ON pg_subscription TO public;
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9ebb026187..d87de2ac2f 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,7 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *binary_given, bool *binary)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +90,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (binary)
+ {
+ *binary_given = false;
+
+ /*
+ * Not all versions of pgoutput will understand this option, so
+ * default to false.
+ */
+ *binary = false;
+ }
/* Parse options */
foreach(lc, options)
@@ -175,6 +185,11 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "binary") == 0 && binary)
+ {
+ *binary_given = true;
+ *binary = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +340,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
char originname[NAMEDATALEN];
bool create_slot;
List *publications;
+ bool binary;
+ bool binary_given;
/*
* Parse and check options.
@@ -334,7 +351,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &binary_given, &binary);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -400,6 +417,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
DirectFunctionCall1(namein, CStringGetDatum(stmt->subname));
values[Anum_pg_subscription_subowner - 1] = ObjectIdGetDatum(owner);
values[Anum_pg_subscription_subenabled - 1] = BoolGetDatum(enabled);
+ values[Anum_pg_subscription_subbinary - 1] = BoolGetDatum(binary);
values[Anum_pg_subscription_subconninfo - 1] =
CStringGetTextDatum(conninfo);
if (slotname)
@@ -669,10 +687,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool binary_given;
+ bool binary;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &binary_given, &binary);
if (slotname_given)
{
@@ -697,6 +718,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (binary_given)
+ {
+ values[Anum_pg_subscription_subbinary - 1] =
+ BoolGetDatum(binary);
+ replaces[Anum_pg_subscription_subbinary - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +736,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +775,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +812,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9bb6..7989b58019 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -423,7 +423,8 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, ", publication_names %s", pubnames_literal);
PQfreemem(pubnames_literal);
pfree(pubnames_str);
-
+ if (options->proto.logical.binary)
+ appendStringInfoString(&cmd, ", binary 'true'");
appendStringInfoChar(&cmd, ')');
}
else
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd171..b91ca2714d 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -17,8 +17,10 @@
#include "catalog/pg_type.h"
#include "libpq/pqformat.h"
#include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
+#include "utils/rel.h"
#include "utils/syscache.h"
/*
@@ -31,7 +33,7 @@
static void logicalrep_write_attrs(StringInfo out, Relation rel);
static void logicalrep_write_tuple(StringInfo out, Relation rel,
- HeapTuple tuple);
+ HeapTuple tuple, bool binary);
static void logicalrep_read_attrs(StringInfo in, LogicalRepRelation *rel);
static void logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple);
@@ -139,7 +141,7 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple, bool binary)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,7 +149,7 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
pq_sendint32(out, RelationGetRelid(rel));
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary);
}
/*
@@ -179,7 +181,7 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
*/
void
logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+ HeapTuple newtuple, bool binary)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -196,26 +198,22 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
pq_sendbyte(out, 'O'); /* old tuple follows */
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary);
}
pq_sendbyte(out, 'N'); /* new tuple follows */
- logicalrep_write_tuple(out, rel, newtuple);
+ logicalrep_write_tuple(out, rel, newtuple, binary);
}
/*
* Read UPDATE from stream.
*/
-LogicalRepRelId
+void
logicalrep_read_update(StringInfo in, bool *has_oldtuple,
LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -240,15 +238,13 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
action);
logicalrep_read_tuple(in, newtup);
-
- return relid;
}
/*
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple, bool binary)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -264,7 +260,7 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
else
pq_sendbyte(out, 'K'); /* old key follows */
- logicalrep_write_tuple(out, rel, oldtuple);
+ logicalrep_write_tuple(out, rel, oldtuple, binary);
}
/*
@@ -272,14 +268,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
*
* Fills the old tuple.
*/
-LogicalRepRelId
+void
logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
{
char action;
- LogicalRepRelId relid;
-
- /* read the relation id */
- relid = pq_getmsgint(in, 4);
/* read and verify action */
action = pq_getmsgbyte(in);
@@ -287,8 +279,6 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
elog(ERROR, "expected action 'O' or 'K', got %c", action);
logicalrep_read_tuple(in, oldtup);
-
- return relid;
}
/*
@@ -437,7 +427,7 @@ logicalrep_read_typ(StringInfo in, LogicalRepTyp *ltyp)
* Write a tuple to the outputstream, in the most efficient format possible.
*/
static void
-logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
+logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binary)
{
TupleDesc desc;
Datum values[MaxTupleAttributeNumber];
@@ -453,6 +443,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
continue;
nliveatts++;
}
+
pq_sendint16(out, nliveatts);
/* try to allocate enough memory from the get-go */
@@ -479,7 +470,7 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
}
else if (att->attlen == -1 && VARATT_IS_EXTERNAL_ONDISK(values[i]))
{
- pq_sendbyte(out, 'u'); /* unchanged toast column */
+ pq_sendbyte(out, LOGICALREP_UNCHANGED);
continue;
}
@@ -488,12 +479,31 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple)
elog(ERROR, "cache lookup failed for type %u", att->atttypid);
typclass = (Form_pg_type) GETSTRUCT(typtup);
- pq_sendbyte(out, 't'); /* 'text' data follows */
+ if (binary &&
+ OidIsValid(typclass->typreceive) &&
+ (att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
+ (att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
+ {
+ bytea *outputbytes;
+ int len;
+
+ pq_sendbyte(out, LOGICALREP_BINARY);
- outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
- pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
- pfree(outputstr);
+ outputbytes = OidSendFunctionCall(typclass->typsend,
+ values[i]);
+ len = VARSIZE(outputbytes) - VARHDRSZ;
+ pq_sendint(out, len, 4); /* length */
+ pq_sendbytes(out, VARDATA(outputbytes), len); /* data */
+ pfree(outputbytes);
+ }
+ else
+ {
+ pq_sendbyte(out, LOGICALREP_TEXT);
+ outputstr = OidOutputFunctionCall(typclass->typoutput, values[i]);
+ pq_sendcountedtext(out, outputstr, strlen(outputstr), false);
+ pfree(outputstr);
+ }
ReleaseSysCache(typtup);
}
}
@@ -512,7 +522,11 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
/* Get number of attributes */
natts = pq_getmsgint(in, 2);
- memset(tuple->changed, 0, sizeof(tuple->changed));
+ tuple->values = palloc(natts * sizeof(StringInfoData *));
+
+ /* default is unchanged */
+ tuple->format = palloc(natts * sizeof(char));
+ memset(tuple->format, LOGICALREP_UNCHANGED, natts * sizeof(char));
/* Read the data */
for (i = 0; i < natts; i++)
@@ -524,25 +538,52 @@ logicalrep_read_tuple(StringInfo in, LogicalRepTupleData *tuple)
switch (kind)
{
case 'n': /* null */
- tuple->values[i] = NULL;
- tuple->changed[i] = true;
- break;
- case 'u': /* unchanged column */
- /* we don't receive the value of an unchanged column */
- tuple->values[i] = NULL;
- break;
- case 't': /* text formatted value */
+ {
+ tuple->values[i] = (StringInfoData *) NULL;
+ tuple->format[i] = LOGICALREP_TEXT;
+ break;
+ }
+ case LOGICALREP_UNCHANGED:
+ {
+ /* we don't receive the value of an unchanged column */
+ tuple->values[i] = (StringInfoData *) NULL;
+ tuple->format[i] = LOGICALREP_UNCHANGED; /* be explicit */
+ break;
+ }
+ case LOGICALREP_BINARY:
{
int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+
+ tuple->format[i] = LOGICALREP_BINARY;
+
+ len = pq_getmsgint(in, 4); /* read length */
- tuple->changed[i] = true;
+ /* and data */
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->cursor = 0;
+ value->maxlen = len;
+ /* not strictly necessary but the docs say it is required */
+ value->data[len] = '\0';
+ tuple->values[i] = value;
+ break;
+ }
+ case LOGICALREP_TEXT:
+ {
+ int len;
+ StringInfoData *value = palloc(sizeof(StringInfoData));
+ tuple->format[i] = LOGICALREP_TEXT;
len = pq_getmsgint(in, 4); /* read length */
/* and data */
- tuple->values[i] = palloc(len + 1);
- pq_copymsgbytes(in, tuple->values[i], len);
- tuple->values[i][len] = '\0';
+ value->data = palloc(len + 1);
+ pq_copymsgbytes(in, value->data, len);
+ value->len = len;
+ value->data[len] = '\0';
+ tuple->values[i] = value;
}
break;
default:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f90a896fc3..844845c473 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -319,13 +319,12 @@ slot_store_error_callback(void *arg)
}
/*
- * Store data in C string form into slot.
- * This is similar to BuildTupleFromCStrings but TupleTableSlot fits our
- * use better.
+ * Store data into slot.
+ * Data can be either text or binary transfer format
*/
static void
-slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
- char **values)
+slot_store_data(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -351,18 +350,42 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
int remoteattnum = rel->attrmap->attnums[i];
if (!att->attisdropped && remoteattnum >= 0 &&
- values[remoteattnum] != NULL)
+ tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
+ {
+ Oid typreceive;
+ Oid typioparam;
+ int cursor;
+
+ cursor = tupleData->values[remoteattnum]->cursor;
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into
+ * proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -387,9 +410,9 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
}
/*
- * Replace selected columns with user data provided as C strings.
- * This is somewhat similar to heap_modify_tuple but also calls the type
- * input functions on the user data.
+ * Replace selected columns with user data provided either as C strings or in
+ * binary format. This is somewhat similar to heap_modify_tuple but also calls
+ * the type input functions on the user data.
* "slot" is filled with a copy of the tuple in "srcslot", with
* columns selected by the "replaces" array replaced with data values
* from "values".
@@ -398,9 +421,9 @@ slot_store_cstrings(TupleTableSlot *slot, LogicalRepRelMapEntry *rel,
* need to materialize "slot" at the end to make it independent of "srcslot".
*/
static void
-slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
- LogicalRepRelMapEntry *rel,
- char **values, bool *replaces)
+slot_modify_data(TupleTableSlot *slot, TupleTableSlot *srcslot,
+ LogicalRepRelMapEntry *rel,
+ LogicalRepTupleData *tupleData)
{
int natts = slot->tts_tupleDescriptor->natts;
int i;
@@ -438,21 +461,44 @@ slot_modify_cstrings(TupleTableSlot *slot, TupleTableSlot *srcslot,
if (remoteattnum < 0)
continue;
- if (!replaces[remoteattnum])
+ if (tupleData->format[remoteattnum] == LOGICALREP_UNCHANGED)
continue;
- if (values[remoteattnum] != NULL)
+ if (tupleData->values[remoteattnum] != NULL)
{
- Oid typinput;
- Oid typioparam;
-
errarg.local_attnum = i;
errarg.remote_attnum = remoteattnum;
- getTypeInputInfo(att->atttypid, &typinput, &typioparam);
- slot->tts_values[i] =
- OidInputFunctionCall(typinput, values[remoteattnum],
- typioparam, att->atttypmod);
+ if (tupleData->format[remoteattnum] == LOGICALREP_BINARY)
+ {
+ Oid typreceive;
+ Oid typioparam;
+ int cursor;
+
+ cursor = tupleData->values[remoteattnum]->cursor;
+
+ getTypeBinaryInputInfo(att->atttypid, &typreceive, &typioparam);
+ slot->tts_values[i] =
+ OidReceiveFunctionCall(typreceive, tupleData->values[remoteattnum],
+ typioparam, att->atttypmod);
+
+ /*
+ * Do not advance the cursor in case we need to re-read this
+ * This saves us from pushing all of this type logic into
+ * proto.c
+ */
+ tupleData->values[remoteattnum]->cursor = cursor;
+ }
+ else
+ {
+ Oid typinput;
+ Oid typioparam;
+
+ getTypeInputInfo(att->atttypid, &typinput, &typioparam);
+ slot->tts_values[i] =
+ OidInputFunctionCall(typinput, tupleData->values[remoteattnum]->data,
+ typioparam, att->atttypmod);
+ }
slot->tts_isnull[i] = false;
errarg.local_attnum = -1;
@@ -618,8 +664,10 @@ apply_handle_insert(StringInfo s)
ensure_transaction();
+ /* read the relation id */
relid = logicalrep_read_insert(s, &newtup);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -641,7 +689,7 @@ apply_handle_insert(StringInfo s)
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, newtup.values);
+ slot_store_data(remoteslot, rel, &newtup);
slot_fill_defaults(rel, estate, remoteslot);
MemoryContextSwitchTo(oldctx);
@@ -733,9 +781,12 @@ apply_handle_update(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
- &newtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_update(s, &has_oldtup, &oldtup,
+ &newtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -765,7 +816,7 @@ apply_handle_update(StringInfo s)
target_rte = list_nth(estate->es_range_table, 0);
for (int i = 0; i < remoteslot->tts_tupleDescriptor->natts; i++)
{
- if (newtup.changed[i])
+ if (newtup.format[i] != LOGICALREP_UNCHANGED)
target_rte->updatedCols = bms_add_member(target_rte->updatedCols,
i + 1 - FirstLowInvalidHeapAttributeNumber);
}
@@ -776,8 +827,8 @@ apply_handle_update(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel,
- has_oldtup ? oldtup.values : newtup.values);
+ slot_store_data(remoteslot, rel,
+ has_oldtup ? &oldtup : &newtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply update to correct partition. */
@@ -831,8 +882,7 @@ apply_handle_update_internal(ResultRelInfo *relinfo,
{
/* Process and store remote tuple in the slot */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_modify_cstrings(remoteslot, localslot, relmapentry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot, localslot, relmapentry, newtup);
MemoryContextSwitchTo(oldctx);
EvalPlanQualSetSlot(&epqstate, remoteslot);
@@ -875,8 +925,11 @@ apply_handle_delete(StringInfo s)
ensure_transaction();
- relid = logicalrep_read_delete(s, &oldtup);
+ /* read the relation id */
+ relid = pq_getmsgint(s, 4);
rel = logicalrep_rel_open(relid, RowExclusiveLock);
+
+ logicalrep_read_delete(s, &oldtup);
if (!should_apply_changes_for_rel(rel))
{
/*
@@ -900,7 +953,7 @@ apply_handle_delete(StringInfo s)
/* Build the search tuple. */
oldctx = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- slot_store_cstrings(remoteslot, rel, oldtup.values);
+ slot_store_data(remoteslot, rel, &oldtup);
MemoryContextSwitchTo(oldctx);
/* For a partitioned table, apply delete to correct partition. */
@@ -1096,9 +1149,9 @@ apply_handle_tuple_routing(ResultRelInfo *relinfo,
if (found)
{
/* Apply the update. */
- slot_modify_cstrings(remoteslot_part, localslot,
- part_entry,
- newtup->values, newtup->changed);
+ slot_modify_data(remoteslot_part, localslot,
+ part_entry,
+ newtup);
MemoryContextSwitchTo(oldctx);
}
else
@@ -2106,6 +2159,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.binary = MySubscription->binary;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 15379e3118..e209ad8aa5 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -20,6 +20,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/builtins.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
@@ -118,11 +119,14 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *binary)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool binary_option_given = false;
+
+ *binary = false;
foreach(lc, options)
{
@@ -138,7 +142,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (protocol_version_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
protocol_version_given = true;
if (!scanint8(strVal(defel->arg), true, &parsed))
@@ -159,7 +163,7 @@ parse_output_parameters(List *options, uint32 *protocol_version,
if (publication_names_given)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
- errmsg("conflicting or redundant options")));
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
publication_names_given = true;
if (!SplitIdentifierString(strVal(defel->arg), ',',
@@ -168,6 +172,19 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "binary") == 0)
+ {
+ if (binary_option_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options, \"%s\" already provided", defel->defname)));
+ binary_option_given = true;
+
+ if (!parse_bool(strVal(defel->arg), binary))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid binary option")));
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -202,7 +219,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &data->binary);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -397,6 +415,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
+
{
HeapTuple tuple = &change->data.tp.newtuple->tuple;
@@ -411,7 +430,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, relation, tuple, data->binary);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +454,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, relation, oldtuple, newtuple, data->binary);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +474,8 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, relation, oldtuple, data->binary);
+
OutputPluginWrite(ctx, true);
}
else
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 3b870c3b17..a7fd29085f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5963,7 +5963,7 @@ describeSubscriptions(const char *pattern, bool verbose)
PGresult *res;
printQueryOpt myopt = pset.popt;
static const bool translate_columns[] = {false, false, false, false,
- false, false};
+ false, false, false};
if (pset.sversion < 100000)
{
@@ -5987,6 +5987,12 @@ describeSubscriptions(const char *pattern, bool verbose)
gettext_noop("Enabled"),
gettext_noop("Publication"));
+ /* Binary mode is only supported in v14 and higher */
+ if (pset.sversion >= 140000)
+ appendPQExpBuffer(&buf,
+ ", subbinary AS \"%s\"\n",
+ gettext_noop("Binary"));
+
if (verbose)
{
appendPQExpBuffer(&buf,
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..e8b3712574 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool subbinary; /* True if the subscription wants the output
+ * plugin to send data in binary */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +76,8 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool binary; /* Indicates if the subscription wants data in
+ * binary format */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561be9..3971f007ae 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -27,13 +27,19 @@
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_UNCHANGED 'u'
+#define LOGICALREP_BINARY 'b'
+#define LOGICALREP_TEXT 't'
+
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
{
- /* column values in text format, or NULL for a null value: */
- char *values[MaxTupleAttributeNumber];
- /* markers for changed/unchanged column values: */
- bool changed[MaxTupleAttributeNumber];
+ /* column values */
+ StringInfoData **values;
+
+ /* markers for changed/unchanged/binary/text */
+ char *format;
+
} LogicalRepTupleData;
typedef uint32 LogicalRepRelId;
@@ -87,17 +93,17 @@ extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+ HeapTuple newtuple, bool binary);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple);
-extern LogicalRepRelId logicalrep_read_update(StringInfo in,
- bool *has_oldtuple, LogicalRepTupleData *oldtup,
- LogicalRepTupleData *newtup);
+ HeapTuple newtuple, bool binary);
+extern void logicalrep_read_update(StringInfo in,
+ bool *has_oldtuple, LogicalRepTupleData *oldtup,
+ LogicalRepTupleData *newtup);
extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
-extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
- LogicalRepTupleData *oldtup);
+ HeapTuple oldtuple, bool binary);
+extern void logicalrep_read_delete(StringInfo in,
+ LogicalRepTupleData *oldtup);
extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
diff --git a/src/include/replication/pgoutput.h b/src/include/replication/pgoutput.h
index 2e8e9daf44..a2d76689ed 100644
--- a/src/include/replication/pgoutput.h
+++ b/src/include/replication/pgoutput.h
@@ -25,6 +25,7 @@ typedef struct PGOutputData
List *publication_names;
List *publications;
+ bool binary;
} PGOutputData;
#endif /* PGOUTPUT_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index c75dcebea0..7cd1458819 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool binary; /* Ask publisher output plugin to use binary */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out
index e7add9d2b8..af6ed982ee 100644
--- a/src/test/regress/expected/subscription.out
+++ b/src/test/regress/expected/subscription.out
@@ -76,10 +76,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar';
ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+-------------+--------------------+-----------------------------
- regress_testsub | regress_subscription_user | f | {testpub} | off | dbname=regress_doesnotexist
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | f | off | dbname=regress_doesnotexist
(1 row)
ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false);
@@ -91,27 +91,27 @@ ERROR: subscription "regress_doesnotexist" does not exist
ALTER SUBSCRIPTION regress_testsub SET (create_slot = false);
ERROR: unrecognized subscription parameter: "create_slot"
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | off | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | off | dbname=regress_doesnotexist2
(1 row)
BEGIN;
ALTER SUBSCRIPTION regress_testsub ENABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | t | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | t | {testpub2,testpub3} | f
(1 row)
ALTER SUBSCRIPTION regress_testsub DISABLE;
\dRs
- List of subscriptions
- Name | Owner | Enabled | Publication
------------------+---------------------------+---------+---------------------
- regress_testsub | regress_subscription_user | f | {testpub2,testpub3}
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary
+-----------------+---------------------------+---------+---------------------+--------
+ regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f
(1 row)
COMMIT;
@@ -126,10 +126,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (synchronous_commit = foobar);
ERROR: invalid value for parameter "synchronous_commit": "foobar"
HINT: Available values: local, remote_write, remote_apply, on, off.
\dRs+
- List of subscriptions
- Name | Owner | Enabled | Publication | Synchronous commit | Conninfo
----------------------+---------------------------+---------+---------------------+--------------------+------------------------------
- regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | local | dbname=regress_doesnotexist2
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+---------------------+---------------------------+---------+---------------------+--------+--------------------+------------------------------
+ regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | local | dbname=regress_doesnotexist2
(1 row)
-- rename back to keep the rest simple
@@ -155,6 +155,22 @@ DROP SUBSCRIPTION IF EXISTS regress_testsub;
NOTICE: subscription "regress_testsub" does not exist, skipping
DROP SUBSCRIPTION regress_testsub; -- fail
ERROR: subscription "regress_testsub" does not exist
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+ERROR: binary requires a Boolean value
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+WARNING: tables were not subscribed, you will have to run ALTER SUBSCRIPTION ... REFRESH PUBLICATION to subscribe the tables
+\dRs+
+ List of subscriptions
+ Name | Owner | Enabled | Publication | Binary | Synchronous commit | Conninfo
+-----------------+---------------------------+---------+-------------+--------+--------------------+-----------------------------
+ regress_testsub | regress_subscription_user | f | {testpub} | t | off | dbname=regress_doesnotexist
+(1 row)
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql
index 9e234ab8b3..835bd05721 100644
--- a/src/test/regress/sql/subscription.sql
+++ b/src/test/regress/sql/subscription.sql
@@ -117,6 +117,18 @@ COMMIT;
DROP SUBSCRIPTION IF EXISTS regress_testsub;
DROP SUBSCRIPTION regress_testsub; -- fail
+-- fail - binary must be boolean
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = foo);
+
+-- now it works
+CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, binary = true);
+
+\dRs+
+
+ALTER SUBSCRIPTION regress_testsub SET (binary = false);
+ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE);
+DROP SUBSCRIPTION regress_testsub;
+
RESET SESSION AUTHORIZATION;
DROP ROLE regress_subscription_user;
DROP ROLE regress_subscription_user2;
diff --git a/src/test/subscription/t/014_binary.pl b/src/test/subscription/t/014_binary.pl
new file mode 100644
index 0000000000..a252f470ef
--- /dev/null
+++ b/src/test/subscription/t/014_binary.pl
@@ -0,0 +1,108 @@
+# Binary mode logical replication test
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+# Create and initialize a publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# Create and initialize subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+my $ddl = qq(
+ CREATE TABLE public.test_numerical (
+ a INTEGER PRIMARY KEY,
+ b NUMERIC,
+ c FLOAT,
+ d BIGINT
+ );
+ CREATE TABLE public.test_arrays (
+ a INTEGER[] PRIMARY KEY,
+ b NUMERIC[],
+ c TEXT[]
+ ););
+
+# Create content on both sides of the replication
+$node_publisher->safe_psql('postgres', $ddl);
+$node_subscriber->safe_psql('postgres', $ddl);
+
+# Configure logical replication
+$node_publisher->safe_psql('postgres',
+ "CREATE PUBLICATION tpub FOR ALL TABLES");
+
+my $publisher_connstring = $node_publisher->connstr . ' dbname=postgres';
+$node_subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION tsub CONNECTION '$publisher_connstring' " .
+ "PUBLICATION tpub WITH (slot_name = tpub_slot, binary = true)");
+
+# Ensure nodes are in sync with eachother
+$node_publisher->wait_for_catchup('tsub');
+$node_subscriber->poll_query_until('postgres',
+ "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('s', 'r');")
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+# Insert some content and make sure it's replicated across
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{1,2,3}', '{1.1, 1.2, 1.3}', '{"one", "two", "three"}'),
+ ('{3,1,2}', '{1.3, 1.1, 1.2}', '{"three", "one", "two"}');
+
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (1, 1.2, 1.3, 10),
+ (2, 2.2, 2.3, 20),
+ (3, 3.2, 3.3, 30);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+my $result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30', 'check replicated data on subscriber');
+
+# Test to reset back to text formatting, and then to binary again
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = false);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_numerical (a, b, c, d) VALUES
+ (4, 4.2, 4.3, 40);
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c, d FROM test_numerical ORDER BY a");
+
+is($result, '1|1.2|1.3|10
+2|2.2|2.3|20
+3|3.2|3.3|30
+4|4.2|4.3|40', 'check replicated data on subscriber');
+
+$node_subscriber->safe_psql('postgres',
+ "ALTER SUBSCRIPTION tsub SET (binary = true);");
+
+$node_publisher->safe_psql('postgres', qq(
+ INSERT INTO public.test_arrays (a, b, c) VALUES
+ ('{2,3,1}', '{1.2, 1.3, 1.1}', '{"two", "three", "one"}');
+ ));
+
+$node_publisher->wait_for_catchup('tsub');
+
+$result = $node_subscriber->safe_psql('postgres',
+ "SELECT a, b, c FROM test_arrays ORDER BY a");
+
+is($result, '{1,2,3}|{1.1,1.2,1.3}|{one,two,three}
+{2,3,1}|{1.2,1.3,1.1}|{two,three,one}
+{3,1,2}|{1.3,1.1,1.2}|{three,one,two}', 'check replicated data on subscriber');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
--
2.21.1 (Apple Git-122.3)
Dave Cramer <davecramer@gmail.com> writes:
On Tue, 14 Jul 2020 at 12:59, Tom Lane <tgl@sss.pgh.pa.us> wrote:
So I started looking through this seriously, and my first question
is why do the docs and code keep saying that "base types" are sent
in binary? Why not just "data"? Are there any cases where we
don't use binary format, if the subscription requests it?
Well for some reason I thought there were some types that did not have send
and receive functions.
There are, but they're all base types, so this terminology is still
unhelpful ;-).
It'd be possible for the sender to send binary for columns it has a
typsend function for, and otherwise send text. However, this only helps
if the receiver has receive functions for all those types; in
cross-version cases they might disagree about which types can be sent
in binary. (Hm ... maybe we could have the receiver verify that it has
typreceive for every column included in its version of the table, before
asking for binary mode?)
regards, tom lane
Hi,
On 2020-07-14 14:08:53 -0400, Dave Cramer wrote:
On Tue, 14 Jul 2020 at 12:59, Tom Lane <tgl@sss.pgh.pa.us> wrote:
So I started looking through this seriously, and my first question
is why do the docs and code keep saying that "base types" are sent
in binary? Why not just "data"? Are there any cases where we
don't use binary format, if the subscription requests it?If there's not a concrete reason to use that terminology,
I'd rather flush it, because it seems confusing.Well for some reason I thought there were some types that did not have send
and receive functions.
There's also send/receive functions that do not work across systems,
unfortunately :(. In particular record and array send functions embed
type oids and their receive functions verify that they match the local
system. Which basically means that if there's any difference in oid
assignment order between two systems that they will not allow to
send/recv such data between them :(.
I suspect that is what that comments might have been referring to?
I've several times suggested that we should remove those type checks in
recv, as they afaict don't provide any actual value. But unfortunately
there hasn't been much response to that. See e.g.
/messages/by-id/20160426001713.hbqdiwvf4mkzkg55@alap3.anarazel.de
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
There's also send/receive functions that do not work across systems,
unfortunately :(. In particular record and array send functions embed
type oids and their receive functions verify that they match the local
system. Which basically means that if there's any difference in oid
assignment order between two systems that they will not allow to
send/recv such data between them :(.
It's not a problem particularly for built-in types, but I agree
there's an issue for extension types.
I've several times suggested that we should remove those type checks in
recv, as they afaict don't provide any actual value. But unfortunately
there hasn't been much response to that. See e.g.
/messages/by-id/20160426001713.hbqdiwvf4mkzkg55@alap3.anarazel.de
Maybe we could compromise by omitting the check if both OIDs are
outside the built-in range?
regards, tom lane
Hi,
On 2020-07-14 19:46:52 -0400, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:
There's also send/receive functions that do not work across systems,
unfortunately :(. In particular record and array send functions embed
type oids and their receive functions verify that they match the local
system. Which basically means that if there's any difference in oid
assignment order between two systems that they will not allow to
send/recv such data between them :(.It's not a problem particularly for built-in types, but I agree
there's an issue for extension types.
I'm not so sure. That's true for builtin types within a single major
version, but not necessarily across major versions. Not that I can
immediately recall cases where we renumbered type oids.
It also assumes that the type specification exactly matches between the
source / target system. It's probably not a great idea to try to use
send/recv for meaningfully different types, but it doesn't seem to crazy
to e.g. allow to e.g. change varchar to text while doing a major version
upgrade over logical rep.
What is the gain in having these checks? recv functions need to be safe
against arbitrary input, so a type crosscheck doesn't buy additional
safety in that regard. Not that a potential attacker couldn't just
change the content anyways?
I've several times suggested that we should remove those type checks in
recv, as they afaict don't provide any actual value. But unfortunately
there hasn't been much response to that. See e.g.
/messages/by-id/20160426001713.hbqdiwvf4mkzkg55@alap3.anarazel.deMaybe we could compromise by omitting the check if both OIDs are
outside the built-in range?
Hm. That'd be a lot better than the current situation. So I'd definitely
go for that if that's what we can agree on.
Greetings,
Andres Freund
Andres Freund <andres@anarazel.de> writes:
What is the gain in having these checks? recv functions need to be safe
against arbitrary input, so a type crosscheck doesn't buy additional
safety in that regard. Not that a potential attacker couldn't just
change the content anyways?
You're confusing security issues with user-friendliness issues.
Detecting that you sent the wrong type via an OID mismatch error
is a lot less painful than trying to figure out why you've got
errors along the line of "incorrect binary data format".
regards, tom lane
Hi,
On 2020-07-14 22:28:48 -0400, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:
What is the gain in having these checks? recv functions need to be safe
against arbitrary input, so a type crosscheck doesn't buy additional
safety in that regard. Not that a potential attacker couldn't just
change the content anyways?You're confusing security issues with user-friendliness issues.
Detecting that you sent the wrong type via an OID mismatch error
is a lot less painful than trying to figure out why you've got
errors along the line of "incorrect binary data format".
An oid mismatch error without knowing what that's about isn't very
helpful either.
How about adding an errcontext that shows the "source type oid", the
target type oid & type name and, for records, the column name of the
target table? That'd make this a lot easier to debug.
Greetings,
Andres Freund
On Tue, 14 Jul 2020 at 22:47, Andres Freund <andres@anarazel.de> wrote:
Hi,
On 2020-07-14 22:28:48 -0400, Tom Lane wrote:
Andres Freund <andres@anarazel.de> writes:
What is the gain in having these checks? recv functions need to be safe
against arbitrary input, so a type crosscheck doesn't buy additional
safety in that regard. Not that a potential attacker couldn't just
change the content anyways?You're confusing security issues with user-friendliness issues.
Detecting that you sent the wrong type via an OID mismatch error
is a lot less painful than trying to figure out why you've got
errors along the line of "incorrect binary data format".An oid mismatch error without knowing what that's about isn't very
helpful either.How about adding an errcontext that shows the "source type oid", the
target type oid & type name and, for records, the column name of the
target table? That'd make this a lot easier to debug.
So looking at how to confirm that the subscriber has receive functions for
all of the types.
AFAICT we don't have that information since the publication determines what
is sent?
This code line 482 in proto.c attempts to limit what is sent in binary. We
could certainly be more restrictive here.
*if* (binary &&
OidIsValid(typclass->typreceive) &&
(att->atttypid < FirstNormalObjectId || typclass->typtype != 'c') &&
(att->atttypid < FirstNormalObjectId || typclass->typelem == InvalidOid))
Dave Cramer
Working through this ... what is the rationale for having changed
the API of logicalrep_read_update? It seems kind of random,
especially since no comparable change was made to
logicalrep_read_insert. If there's actually a good reason,
it seems like it'd apply to both. If there's not, I'd be
inclined to not change the API, because this sort of thing
is a recipe for bugs when making cross-version patches.
regards, tom lane
I've pushed this patch, with a number of adjustments, some cosmetic
and some not so much (no pg_dump support!?). We're not quite
done though ...
Dave Cramer <davecramer@gmail.com> writes:
So looking at how to confirm that the subscriber has receive functions for
all of the types.
AFAICT we don't have that information since the publication determines what
is sent?
Yeah, at the point where we need to send the option, we seem not to have a
lot of info. In practice, if the sender has a typsend function, the only
way the subscriber doesn't have a matching typreceive function is if it's
an older PG version. I think it's sufficient to document that you can't
use binary mode in that case, so that's what I did. (Note that
getTypeBinaryInputInfo will say "no binary input function available for
type %s" in such a case, so that seemed like adequate error handling.)
On Tue, 14 Jul 2020 at 22:47, Andres Freund <andres@anarazel.de> wrote:
An oid mismatch error without knowing what that's about isn't very
helpful either.
How about adding an errcontext that shows the "source type oid", the
target type oid & type name and, for records, the column name of the
target table? That'd make this a lot easier to debug.
This code line 482 in proto.c attempts to limit what is sent in binary. We
could certainly be more restrictive here.
I think Andres' point is to be *less* restrictive. I left that logic
as-is in the committed patch, but we could do something like the attached
to improve the situation.
regards, tom lane
Attachments:
relax-array-and-record-OID-checks.patchtext/x-diff; charset=us-ascii; name=relax-array-and-record-OID-checks.patchDownload
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 2b1356ee24..d9837a5971 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -494,22 +494,9 @@ logicalrep_write_tuple(StringInfo out, Relation rel, HeapTuple tuple, bool binar
typclass = (Form_pg_type) GETSTRUCT(typtup);
/*
- * Choose whether to send in binary. Obviously, the option must be
- * requested and the type must have a send function. Also, if the
- * type is not built-in then it must not be a composite or array type.
- * Such types contain type OIDs, which will likely not match at the
- * receiver if it's not a built-in type.
- *
- * XXX this could be relaxed if we changed record_recv and array_recv
- * to be less picky.
- *
- * XXX this fails to apply the restriction to domains over such types.
+ * Send in binary if requested and type has suitable send function.
*/
- if (binary &&
- OidIsValid(typclass->typsend) &&
- (att->atttypid < FirstGenbkiObjectId ||
- (typclass->typtype != TYPTYPE_COMPOSITE &&
- typclass->typelem == InvalidOid)))
+ if (binary && OidIsValid(typclass->typsend))
{
bytea *outputbytes;
int len;
diff --git a/src/backend/utils/adt/arrayfuncs.c b/src/backend/utils/adt/arrayfuncs.c
index 800107d4e7..392445ea03 100644
--- a/src/backend/utils/adt/arrayfuncs.c
+++ b/src/backend/utils/adt/arrayfuncs.c
@@ -1308,13 +1308,34 @@ array_recv(PG_FUNCTION_ARGS)
(errcode(ERRCODE_INVALID_BINARY_REPRESENTATION),
errmsg("invalid array flags")));
+ /* Check element type recorded in the data */
element_type = pq_getmsgint(buf, sizeof(Oid));
+
+ /*
+ * From a security standpoint, it doesn't matter whether the input's
+ * element type matches what we expect: the element type's receive
+ * function has to be robust enough to cope with invalid data. However,
+ * from a user-friendliness standpoint, it's nicer to complain about type
+ * mismatches than to throw "improper binary format" errors. But there's
+ * a problem: only built-in types have OIDs that are stable enough to
+ * believe that a mismatch is a real issue. So complain only if both OIDs
+ * are in the built-in range. Otherwise, carry on with the element type
+ * we "should" be getting.
+ */
if (element_type != spec_element_type)
{
- /* XXX Can we allow taking the input element type in any cases? */
- ereport(ERROR,
- (errcode(ERRCODE_DATATYPE_MISMATCH),
- errmsg("wrong element type")));
+ if (element_type < FirstGenbkiObjectId &&
+ spec_element_type < FirstGenbkiObjectId)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("binary data has array element type %u (%s) instead of expected %u (%s)",
+ element_type,
+ format_type_extended(element_type, -1,
+ FORMAT_TYPE_ALLOW_INVALID),
+ spec_element_type,
+ format_type_extended(spec_element_type, -1,
+ FORMAT_TYPE_ALLOW_INVALID))));
+ element_type = spec_element_type;
}
for (i = 0; i < ndim; i++)
diff --git a/src/backend/utils/adt/rowtypes.c b/src/backend/utils/adt/rowtypes.c
index 80cba2f4c2..674cf0a55d 100644
--- a/src/backend/utils/adt/rowtypes.c
+++ b/src/backend/utils/adt/rowtypes.c
@@ -551,13 +551,33 @@ record_recv(PG_FUNCTION_ARGS)
continue;
}
- /* Verify column datatype */
+ /* Check column type recorded in the data */
coltypoid = pq_getmsgint(buf, sizeof(Oid));
- if (coltypoid != column_type)
+
+ /*
+ * From a security standpoint, it doesn't matter whether the input's
+ * column type matches what we expect: the column type's receive
+ * function has to be robust enough to cope with invalid data.
+ * However, from a user-friendliness standpoint, it's nicer to
+ * complain about type mismatches than to throw "improper binary
+ * format" errors. But there's a problem: only built-in types have
+ * OIDs that are stable enough to believe that a mismatch is a real
+ * issue. So complain only if both OIDs are in the built-in range.
+ * Otherwise, carry on with the column type we "should" be getting.
+ */
+ if (coltypoid != column_type &&
+ coltypoid < FirstGenbkiObjectId &&
+ column_type < FirstGenbkiObjectId)
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
- errmsg("wrong data type: %u, expected %u",
- coltypoid, column_type)));
+ errmsg("binary data has type %u (%s) instead of expected %u (%s) in record column %d",
+ coltypoid,
+ format_type_extended(coltypoid, -1,
+ FORMAT_TYPE_ALLOW_INVALID),
+ column_type,
+ format_type_extended(column_type, -1,
+ FORMAT_TYPE_ALLOW_INVALID),
+ i + 1)));
/* Get and check the item length */
itemlen = pq_getmsgint(buf, 4);
On Sat, Jul 18, 2020 at 9:53 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
I've pushed this patch, with a number of adjustments, some cosmetic
and some not so much (no pg_dump support!?). We're not quite
done though ...
Skink's latest run reports a failure that I surmise was caused by this patch:
==722318== VALGRINDERROR-BEGIN
==722318== Invalid read of size 1
==722318== at 0x4F4CC9: apply_handle_update (worker.c:834)
==722318== by 0x4F4F81: apply_dispatch (worker.c:1427)
==722318== by 0x4F5104: LogicalRepApplyLoop (worker.c:1635)
==722318== by 0x4F57BF: ApplyWorkerMain (worker.c:2141)
==722318== by 0x4BD49E: StartBackgroundWorker (bgworker.c:813)
==722318== by 0x4CBAB4: do_start_bgworker (postmaster.c:5865)
==722318== by 0x4CBBF5: maybe_start_bgworkers (postmaster.c:6091)
==722318== by 0x4CC4BF: sigusr1_handler (postmaster.c:5260)
==722318== by 0x486413F: ??? (in
/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
==722318== by 0x4DC7845: select (select.c:41)
==722318== by 0x4CCE40: ServerLoop (postmaster.c:1691)
==722318== by 0x4CE106: PostmasterMain (postmaster.c:1400)
==722318== Address 0x78cb0ab is 443 bytes inside a recently
re-allocated block of size 8,192 alloc'd
==722318== at 0x483877F: malloc (vg_replace_malloc.c:307)
==722318== by 0x6A55BD: AllocSetContextCreateInternal (aset.c:468)
==722318== by 0x280262: AtStart_Memory (xact.c:1108)
==722318== by 0x2806ED: StartTransaction (xact.c:1979)
==722318== by 0x282128: StartTransactionCommand (xact.c:2829)
==722318== by 0x4F5514: ApplyWorkerMain (worker.c:2014)
==722318== by 0x4BD49E: StartBackgroundWorker (bgworker.c:813)
==722318== by 0x4CBAB4: do_start_bgworker (postmaster.c:5865)
==722318== by 0x4CBBF5: maybe_start_bgworkers (postmaster.c:6091)
==722318== by 0x4CC4BF: sigusr1_handler (postmaster.c:5260)
==722318== by 0x486413F: ??? (in
/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
==722318== by 0x4DC7845: select (select.c:41)
==722318==
==722318== VALGRINDERROR-END
See https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2020-07-20%2002%3A37%3A51
--
Peter Geoghegan
Peter Geoghegan <pg@bowt.ie> writes:
Skink's latest run reports a failure that I surmise was caused by this patch:
Yeah, I've just been digging through that. The patch didn't create
the bug, but it allowed valgrind to detect it, because the column
status array is now "just big enough" rather than being always
MaxTupleAttributeNumber entries. To wit, the problem is that the
code in apply_handle_update that computes target_rte->updatedCols
is junk.
The immediate issue is that it fails to apply the remote-to-local
column number mapping, so that it's looking at the wrong colstatus
entries, possibly including entries past the end of the array.
I'm fixing that, but even after that, there's a semantic problem:
LOGICALREP_COLUMN_UNCHANGED is just a weak optimization, cf the code
that sends it, in proto.c around line 480. colstatus will often *not*
be that for columns that were in fact not updated on the remote side.
I wonder whether we need to take steps to improve that.
CC'ing Peter E., as this issue arose with b9c130a1fdf.
regards, tom lane
Hi,
On 20/07/2020 17:51, Tom Lane wrote:
Peter Geoghegan <pg@bowt.ie> writes:
Skink's latest run reports a failure that I surmise was caused by this patch:
Yeah, I've just been digging through that. The patch didn't create
the bug, but it allowed valgrind to detect it, because the column
status array is now "just big enough" rather than being always
MaxTupleAttributeNumber entries. To wit, the problem is that the
code in apply_handle_update that computes target_rte->updatedCols
is junk.The immediate issue is that it fails to apply the remote-to-local
column number mapping, so that it's looking at the wrong colstatus
entries, possibly including entries past the end of the array.I'm fixing that, but even after that, there's a semantic problem:
LOGICALREP_COLUMN_UNCHANGED is just a weak optimization, cf the code
that sends it, in proto.c around line 480. colstatus will often *not*
be that for columns that were in fact not updated on the remote side.
I wonder whether we need to take steps to improve that.
LOGICALREP_COLUMN_UNCHANGED is not trying to optimize anything, there is
certainly no effort made to not send columns that were not updated by
logical replication itself. It's just something we invented in order to
handle the fact that values for TOASTed columns that were not updated
are simply not visible to logical decoding (unless table has REPLICA
IDENTITY FULL) as they are not written to WAL nor accessible via
historic snapshot. So the output plugin simply does not see the real value.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Petr Jelinek <petr@2ndquadrant.com> writes:
On 20/07/2020 17:51, Tom Lane wrote:
I'm fixing that, but even after that, there's a semantic problem:
LOGICALREP_COLUMN_UNCHANGED is just a weak optimization, cf the code
that sends it, in proto.c around line 480. colstatus will often *not*
be that for columns that were in fact not updated on the remote side.
I wonder whether we need to take steps to improve that.
LOGICALREP_COLUMN_UNCHANGED is not trying to optimize anything, there is
certainly no effort made to not send columns that were not updated by
logical replication itself. It's just something we invented in order to
handle the fact that values for TOASTed columns that were not updated
are simply not visible to logical decoding (unless table has REPLICA
IDENTITY FULL) as they are not written to WAL nor accessible via
historic snapshot. So the output plugin simply does not see the real value.
Hm. So the comment I added a couple days ago is wrong; can you propose
a better one?
However, be that as it may, we do have a provision in the protocol that
can handle marking columns unchanged. I'm thinking if we tried a bit
harder to identify unchanged columns on the sending side, we could both
fix this semantic deficiency for triggers and improve efficiency by
reducing transmission of unneeded data.
regards, tom lane