BUG #14319: Logical decoding dropping statements in subtransactions
The following bug has been logged on the website:
Bug reference: 14319
Logged by: Huan Ruan
Email address: leohuanruan@gmail.com
PostgreSQL version: 9.5.4
Operating system: CentOS 7.2.1511 x86_64
Description:
When inserting records within a subtransaction, not all inserts are reported
by pg_logical_slot_peek_changes(). With more than 4096 inserts, the output
is truncated to the nearest multiple of 4096.
This occurs in 9.5.4, but not in 9.5.3.
To reproduce:
/usr/pgsql-9.5/bin/initdb
echo 'wal_level=logical' >> $PGDATA/postgresql.conf
echo 'max_replication_slots=1' >> $PGDATA/postgresql.conf
/usr/pgsql-9.5/bin/pg_ctl start -w
createdb test1
psql test1 <<EOF
SELECT pg_create_logical_replication_slot('regression_slot',
'test_decoding');
CREATE TABLE t (x BIGINT);
BEGIN;
SAVEPOINT s;
INSERT INTO t SELECT generate_series(1, 5000);
RELEASE SAVEPOINT s;
COMMIT;
/* Reports 4096 inserts, rather than the expected 5000 */
SELECT COUNT(*) FROM pg_logical_slot_peek_changes('regression_slot', NULL,
NULL) WHERE data ~ 'INSERT';
EOF
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Fri, Sep 9, 2016 at 10:26 AM, <leohuanruan@gmail.com> wrote:
When inserting records within a subtransaction, not all inserts are reported
by pg_logical_slot_peek_changes(). With more than 4096 inserts, the output
is truncated to the nearest multiple of 4096.This occurs in 9.5.4, but not in 9.5.3.
And the first commit to cause this regression is this one:
commit: 8caf9fe62544b351d4f6219bf416f5ce08ef3c21
author: Tom Lane <tgl@sss.pgh.pa.us>
date: Thu, 30 Jun 2016 12:37:02 -0400
Fix typo in ReorderBufferIterTXNInit().
This looks like it would cause changes from subtransactions to be missed
by the iterator being constructed, if those changes had been spilled to
disk previously. This implies that large subtransactions might be lost
(in whole or in part) by logical replication. Found and fixed by
Petru-Florin Mihancea, per bug #14208.
Report: <20160622144830.5791.22512@wrigleys.postgresql.org>
4096 matches the maximum number of changes that can be saved in memory
from reorderbuffer.c. Andres?
--
Michael
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
Thanks Michael for pinpointing the commit. Can I take it that this is a
confirmed bug? If so, how likely is it to be fixed in 9.5?
On 17:33, Tue, 13/09/2016 Michael Paquier <michael.paquier@gmail.com> wrote:
Show quoted text
On Fri, Sep 9, 2016 at 10:26 AM, <leohuanruan@gmail.com> wrote:
When inserting records within a subtransaction, not all inserts are
reported
by pg_logical_slot_peek_changes(). With more than 4096 inserts, the
output
is truncated to the nearest multiple of 4096.
This occurs in 9.5.4, but not in 9.5.3.
And the first commit to cause this regression is this one:
commit: 8caf9fe62544b351d4f6219bf416f5ce08ef3c21
author: Tom Lane <tgl@sss.pgh.pa.us>
date: Thu, 30 Jun 2016 12:37:02 -0400
Fix typo in ReorderBufferIterTXNInit().This looks like it would cause changes from subtransactions to be missed
by the iterator being constructed, if those changes had been spilled to
disk previously. This implies that large subtransactions might be lost
(in whole or in part) by logical replication. Found and fixed by
Petru-Florin Mihancea, per bug #14208.Report: <20160622144830.5791.22512@wrigleys.postgresql.org>
4096 matches the maximum number of changes that can be saved in memory
from reorderbuffer.c. Andres?
--
Michael
On Thu, Sep 15, 2016 at 5:32 PM, Huan Ruan <leohuanruan@gmail.com> wrote:
Thanks Michael for pinpointing the commit. Can I take it that this is a
confirmed bug? If so, how likely is it to be fixed in 9.5?
(Please avoid top-posting)
To me this is really a bug, and deserves a fix down to 9.4 where
logical decoding is supported. It just needs some care.
--
Michael
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Thu, 15 Sep 2016 at 18:52, Michael Paquier <michael.paquier@gmail.com>
wrote:
On Thu, Sep 15, 2016 at 5:32 PM, Huan Ruan <leohuanruan@gmail.com> wrote:
Thanks Michael for pinpointing the commit. Can I take it that this is a
confirmed bug? If so, how likely is it to be fixed in 9.5?(Please avoid top-posting)
To me this is really a bug, and deserves a fix down to 9.4 where
logical decoding is supported. It just needs some care.
--
Michael
Thanks again.
Hi
Please can someone update me when can we expect an attempt to fix this bug?
This behavior, silently missing data, is a serious issue. Without fixing
it, I don't believe logical encoding can be used.
Regards
Huan
On September 19, 2016 6:21:13 AM PDT, Huan Ruan <leohuanruan@gmail.com> wrote:
Hi
Please can someone update me when can we expect an attempt to fix this
bug?
This behavior, silently missing data, is a serious issue. Without
fixing
it, I don't believe logical encoding can be used.
I'll new looking into the issue.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Tue, 20 Sep 2016 at 04:53 Andres Freund <andres@anarazel.de> wrote:
I'll new looking into the issue.
Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Thanks Andres.
On 2016-09-20 00:53:49 +0000, Huan Ruan wrote:
On Tue, 20 Sep 2016 at 04:53 Andres Freund <andres@anarazel.de> wrote:
I'll new looking into the issue.
Thanks Andres.
Found the issue, and a fix. Wrote a good number of tests. Too tired to
commit tonight, will do so tomorrow.
The issue isn't actually related to the commit Michael pointed out. I
don't think the results were correct before the last release either :(
Attached the patch, but it needs some minor amount of polish.
Greetings,
Andres Freund
Attachments:
0001-Correct-logical-decoding-restore-behaviour-for-subtr.patchtext/x-patch; charset=us-asciiDownload+445-7
On Mon, 3 Oct 2016, 19:16 Andres Freund <andres@anarazel.de> wrote:
Found the issue, and a fix. Wrote a good number of tests. Too tired to
commit tonight, will do so tomorrow.The issue isn't actually related to the commit Michael pointed out. I
don't think the results were correct before the last release either :(Attached the patch, but it needs some minor amount of polish.
Greetings,
Andres Freund
That's great! Many thanks again Andres. Really appreciate it.
Regards
Huan
Show quoted text
On 2016-10-03 08:28:46 +0000, Huan Ruan wrote:
On Mon, 3 Oct 2016, 19:16 Andres Freund <andres@anarazel.de> wrote:
Found the issue, and a fix. Wrote a good number of tests. Too tired to
commit tonight, will do so tomorrow.The issue isn't actually related to the commit Michael pointed out. I
don't think the results were correct before the last release either :(Attached the patch, but it needs some minor amount of polish.
Greetings,
Andres Freund
That's great! Many thanks again Andres. Really appreciate it.
And pushed (9.4 - master). Will be contained in the next set of
minor releases.
Regards,
Andres
--
Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-bugs
On Tue, 4 Oct 2016, 16:20 Andres Freund <andres@anarazel.de> wrote:
On 2016-10-03 08:28:46 +0000, Huan Ruan wrote:
On Mon, 3 Oct 2016, 19:16 Andres Freund <andres@anarazel.de> wrote
And pushed (9.4 - master). Will be contained in the next set of
minor releases.Regards,
Andres
Great. Looking forward to it. Thank you Andres.
Regards
Huan