"snapshot too large" error when initializing logical replication (9.4)
I sometimes get the error "snapshot too large" from my logical
replication walsender process when in response to a CREATE_REPLICATION_SLOT.
This is in SnapBuildExportSnapshot in snapbuild.c
newxcnt is 212 at that point
I have max_connections = 200
procArray->maxProcs=212
Should we be testing
newxcnt > GetMaxSnapshotXidCount()
instead of
newxcnt >= GetMaxSnapshotXidCount()
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.
Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.
I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.
This is in SnapBuildExportSnapshot in snapbuild.c
newxcnt is 212 at that point
I have max_connections = 200
procArray->maxProcs=212
Should we be testing
newxcnt > GetMaxSnapshotXidCount()instead of
newxcnt >= GetMaxSnapshotXidCount()
It actually looks correct to me new - newxcnt is used as an offset into
an array of size GetMaxSnapshotXidCount().
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
larger-snapbuildexportsnapshot-xid-array.difftext/x-diff; charset=us-asciiDownload
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 5e59c6b..2df1905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -519,6 +519,7 @@ SnapBuildExportSnapshot(SnapBuild *builder)
TransactionId xid;
TransactionId *newxip;
int newxcnt = 0;
+ int newxcnt_max;
if (builder->state != SNAPBUILD_CONSISTENT)
elog(ERROR, "cannot export a snapshot before reaching a consistent state");
@@ -557,8 +558,9 @@ SnapBuildExportSnapshot(SnapBuild *builder)
MyPgXact->xmin = snap->xmin;
/* allocate in transaction context */
+ newxcnt_max = GetMaxSnapshotXidCount();
newxip = (TransactionId *)
- palloc(sizeof(TransactionId) * GetMaxSnapshotXidCount());
+ palloc(sizeof(TransactionId) * newxcnt_max);
/*
* snapbuild.c builds transactions in an "inverted" manner, which means it
@@ -579,8 +581,11 @@ SnapBuildExportSnapshot(SnapBuild *builder)
if (test == NULL)
{
- if (newxcnt >= GetMaxSnapshotXidCount())
- elog(ERROR, "snapshot too large");
+ if (newxcnt >= newxcnt_max)
+ {
+ newxcnt_max *= 2;
+ newxip = repalloc(newxip, sizeof(TransactionId) * newxcnt_max);
+ }
newxip[newxcnt++] = xid;
}
On 10/28/2014 01:27 PM, Andres Freund wrote:
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.
This patch seems to fix things.
I've done numerous runs of the test with I was doing before with your
patch applied and don't seem to be having this issue anymore.
This is in SnapBuildExportSnapshot in snapbuild.c
newxcnt is 212 at that point
I have max_connections = 200
procArray->maxProcs=212
Should we be testing
newxcnt > GetMaxSnapshotXidCount()instead of
newxcnt >= GetMaxSnapshotXidCount()It actually looks correct to me new - newxcnt is used as an offset into
an array of size GetMaxSnapshotXidCount().Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Andres Freund wrote:
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.
You haven't pushed this, have you?
--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
Andres Freund wrote:
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.You haven't pushed this, have you?
No, but it's on my todo list.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote:
On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
Andres Freund wrote:
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.You haven't pushed this, have you?
No, but it's on my todo list.
Uh, where are we on this?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2015-03-20 08:41:38 -0400, Bruce Momjian wrote:
On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote:
On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote:
Andres Freund wrote:
Hi,
On 2014-10-25 18:09:36 -0400, Steve Singer wrote:
I sometimes get the error "snapshot too large" from my logical replication
walsender process when in response to a CREATE_REPLICATION_SLOT.Yes. That's possible if 'too much' was going on until a consistent point
was reached. I think we can just use a much larger size for the array
if necessary.I've attached patch for this. Could you try whether that helps? I don't
have a testcase handy that reproduces the problem.You haven't pushed this, have you?
No, but it's on my todo list.
Uh, where are we on this?
Nowhere, unfortunately. The fix I'd developed isn't actually
correct. The problem is that we can only import a transaction with fewer
than GetMaxSnapshotXidCount() transactions atm. The only reason it
worked in Steve's case is because he probably wasn't importin gthe
snapshot in the test case...
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers