Logical replication - TRAP: FailedAssertion in pgstat.c
Testing logical replication, with the following patches on top of
yesterday's master:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patch
Is applying that patch set is still correct?
It builds fine, but when I run the old pbench-over-logical-replication
test I get:
TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)
reliably (often within a minute).
The test itself does not fail, at least not that I saw (but I only ran a
few).
thanks,
Erik Rijkers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patch
I am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.com
It builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)
To get that error:
--------------
#!/bin/sh
port1=6972 port2=6973 scale=25 clients=16 duration=60
echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port1 \
&& echo "drop table if exists pgbench_accounts;
drop table if exists pgbench_branches;
drop table if exists pgbench_tellers;
drop table if exists pgbench_history;" | psql -qXp $port2 \
&& pgbench -p $port1 -qis ${scale//_/} && echo "
alter table pgbench_history add column hid serial primary key;
" | psql -q1Xp $port1 \
&& pg_dump -F c -p $port1 \
--exclude-table-data=pgbench_history \
--exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches \
--exclude-table-data=pgbench_tellers \
-t pgbench_history \
-t pgbench_accounts \
-t pgbench_branches \
-t pgbench_tellers \
| pg_restore -1 -p $port2 -d testdb
appname=pgbench_derail
echo "create publication pub1 for all tables;" | psql -p $port1 -aqtAX
echo "create subscription sub1 connection 'port=${port1}
application_name=${appname}' publication pub1 with (disabled);
alter subscription sub1 enable;
" | psql -p $port2 -aqtAX
echo "-- pgbench -p $port1 -c $clients -T $duration -n -- scale $scale
"
pgbench -p $port1 -c $clients -T $duration -n
--------------
Erik Rijkers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patchI am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.comIt builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 16/04/17 20:41, Andres Freund wrote:
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patchI am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.comIt builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?- Andres
Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though) or put
it into it's own if that checks if tx was indeed committed (which is bit
ugly).
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 2017-04-16 20:41, Andres Freund wrote:
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patchI am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.comIt builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?
Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
flawlessly again. (flawlessly= a few hours without any error)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 17 Apr 2017, at 10:30, Erik Rijkers <er@xs4all.nl> wrote:
On 2017-04-16 20:41, Andres Freund wrote:
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patchI am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.comIt builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?Yes, compiled from 67c2def11d4 with the above 4 patches, it runs flawlessly again. (flawlessly= a few hours without any error)
I’ve reproduced failure, this happens under tablesync worker and putting
pgstat_report_stat() under the previous condition block should help.
However for me it took about an hour of running this script to catch original assert.
Can you check with that patch applied?
Attachments:
logical_worker.patchapplication/octet-stream; name=logical_worker.patch; x-unix-mode=0644Download
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 656d399..7a02c33 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -453,6 +453,7 @@ apply_handle_commit(StringInfo s)
replorigin_session_origin_timestamp = commit_data.committime;
CommitTransactionCommand();
+ pgstat_report_stat(false);
store_flush_position(commit_data.end_lsn);
}
@@ -462,7 +463,6 @@ apply_handle_commit(StringInfo s)
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(commit_data.end_lsn);
- pgstat_report_stat(false);
pgstat_report_activity(STATE_IDLE, NULL);
}
On 2017-04-17 15:59, Stas Kelvich wrote:
On 17 Apr 2017, at 10:30, Erik Rijkers <er@xs4all.nl> wrote:
On 2017-04-16 20:41, Andres Freund wrote:
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote:
On 2017-04-15 04:47, Erik Rijkers wrote:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patchI am now using these newer patches:
/messages/by-id/30242bc6-eca4-b7bb-670e-8d0458753a8c@2ndquadrant.comIt builds fine, but when I run the old pbench-over-logical-replication
test I get:TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)To get that error:
I presume this is the fault of
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746
if you git revert that individual commit, do things work again?Yes, compiled from 67c2def11d4 with the above 4 patches, it runs
flawlessly again. (flawlessly= a few hours without any error)I’ve reproduced failure, this happens under tablesync worker and
putting
pgstat_report_stat() under the previous condition block should help.However for me it took about an hour of running this script to catch
original assert.Can you check with that patch applied?
Your patch on top of the 5 patches above seem to solve the matter too:
no problems after running for 2 hours (previously it failed within half
a minute).
Erik Rijkers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 4/16/17 16:11, Petr Jelinek wrote:
Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though)
I think to avoid the latter, we should add more pgstat_report_stat()
calls, such as in process_syncing_tables_for_apply(). Basically every
code path that calls CommitTransactionCommand() should have one, no?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
On 4/16/17 16:11, Petr Jelinek wrote:
Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though)I think to avoid the latter, we should add more pgstat_report_stat()
calls, such as in process_syncing_tables_for_apply(). Basically every
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:On 4/16/17 16:11, Petr Jelinek wrote:
Yeah it is, it needs to be fenced to happen only after commit, which is
not guaranteed at the point of code, we probably need to put the
pgstat_report_stat() inside the if above after the
CommitTransactionCommand() (that will make it report stats for changes
apply did to pg_subscription_rel after next transaction though)I think to avoid the latter, we should add more pgstat_report_stat()
calls, such as in process_syncing_tables_for_apply(). Basically every
code path that calls CommitTransactionCommand() should have one, no?Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Attachments:
fix-statistics-reporting-in-logical-replication-work.patchbinary/octet-stream; name=fix-statistics-reporting-in-logical-replication-work.patchDownload
From 0dcecaffb7ba94b900bbd5a483ca2cdfa996a952 Mon Sep 17 00:00:00 2001
From: Petr Jelinek <pjmodos@pjmodos.net>
Date: Wed, 3 May 2017 07:35:41 +0200
Subject: [PATCH] fix statistics reporting in logical replication workers
---
src/backend/replication/logical/tablesync.c | 18 ++++++++++++++----
src/backend/replication/logical/worker.c | 2 +-
2 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 0823000..cf331c0 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -274,6 +274,7 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
static List *table_states = NIL;
static HTAB *last_start_times = NULL;
ListCell *lc;
+ bool xact_started = false;
Assert(!IsTransactionState());
@@ -290,6 +291,7 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
table_states = NIL;
StartTransactionCommand();
+ xact_started = true;
/* Fetch all non-ready tables. */
rstates = GetSubscriptionNotReadyRelations(MySubscription->oid);
@@ -304,8 +306,6 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
}
MemoryContextSwitchTo(oldctx);
- CommitTransactionCommand();
-
table_states_valid = true;
}
@@ -350,11 +350,14 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
{
rstate->state = SUBREL_STATE_READY;
rstate->lsn = current_lsn;
- StartTransactionCommand();
+ if (!xact_started)
+ {
+ StartTransactionCommand();
+ xact_started = true;
+ }
SetSubscriptionRelState(MyLogicalRepWorker->subid,
rstate->relid, rstate->state,
rstate->lsn);
- CommitTransactionCommand();
}
}
else
@@ -457,6 +460,12 @@ process_syncing_tables_for_apply(XLogRecPtr current_lsn)
}
}
}
+
+ if (xact_started)
+ {
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+ }
}
/*
@@ -806,6 +815,7 @@ LogicalRepSyncTableStart(XLogRecPtr *origin_startpos)
MyLogicalRepWorker->relstate,
MyLogicalRepWorker->relstate_lsn);
CommitTransactionCommand();
+ pgstat_report_stat(false);
/*
* We want to do the table data sync in single
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 692fa3a..eb3d99b 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -451,6 +451,7 @@ apply_handle_commit(StringInfo s)
replorigin_session_origin_timestamp = commit_data.committime;
CommitTransactionCommand();
+ pgstat_report_stat(false);
store_flush_position(commit_data.end_lsn);
}
@@ -460,7 +461,6 @@ apply_handle_commit(StringInfo s)
/* Process any tables that are being synchronized in parallel. */
process_syncing_tables(commit_data.end_lsn);
- pgstat_report_stat(false);
pgstat_report_activity(STATE_IDLE, NULL);
}
--
2.7.4
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.
I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.
With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):
fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not see
the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)
thanks
Erik Rijkers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Wed, May 03, 2017 at 01:23:19PM +0200, Erik Rijkers wrote:
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.I'm running my pgbench-over-logical-replication test in chunk of 15 minutes,
wth different pgbench -c (num clients) and -s (scale) values.With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it seems
because of "inital snapshot too large") but indeed I do not see the 'TRAP:
FailedAssertion in pgstat.c' anymore.(If there is any other configuration of patches worth testing please let me
know)
[Action required within three days. This is a generic notification.]
The above-described topic is currently a PostgreSQL 10 open item. Peter,
since you committed the patch believed to have created it, you own this open
item. If some other commit is more relevant or if this does not belong as a
v10 open item, please let us know. Otherwise, please observe the policy on
open item ownership[1]/messages/by-id/20170404140717.GA2675809@tornado.leadboat.com and send a status update within three calendar days of
this message. Include a date for your subsequent status update. Testers may
discover new open items at any time, and I want to plan to get them all fixed
well in advance of shipping v10. Consequently, I will appreciate your efforts
toward speedy resolution. Thanks.
[1]: /messages/by-id/20170404140717.GA2675809@tornado.leadboat.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 03/05/17 13:23, Erik Rijkers wrote:
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not see
Yes that's different thing that we've been discussing a bit in snapbuild
woes thread.
the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)
Thanks, so the patch works.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
<petr.jelinek@2ndquadrant.com> wrote:
On 03/05/17 13:23, Erik Rijkers wrote:
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not seeYes that's different thing that we've been discussing a bit in snapbuild
woes thread.the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)Thanks, so the patch works.
I think that we should commit the local transaction that did initial
data copy, and then report stat as well. Currently table sync worker
doesn't commit the local transaction in LogicalRepSyncTableStart
(maybe until apply commit record?) if its status is changed to
SUBREL_STATE_CATCHUP. That's why the table sync worker issues
assertion failure.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 5/5/17 07:13, Petr Jelinek wrote:
Yes that's different thing that we've been discussing a bit in snapbuild
woes thread.the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)Thanks, so the patch works.
Committed, with s/xact_started/started_tx/, to match nearby code.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 08/05/17 17:52, Masahiko Sawada wrote:
On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
<petr.jelinek@2ndquadrant.com> wrote:On 03/05/17 13:23, Erik Rijkers wrote:
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not seeYes that's different thing that we've been discussing a bit in snapbuild
woes thread.the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)Thanks, so the patch works.
I think that we should commit the local transaction that did initial
data copy, and then report stat as well. Currently table sync worker
doesn't commit the local transaction in LogicalRepSyncTableStart
(maybe until apply commit record?) if its status is changed to
SUBREL_STATE_CATCHUP. That's why the table sync worker issues
assertion failure.
That would fix the assert as well yes, but it would also mean that if
the worker crashed between the initial copy and the end of catchup there
would be no way to restart it without manual intervention from user
since the synchronization position would be lost. Hence the fix I
proposed which does it differently and has the whole sync in a single
transaction.
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 08/05/17 18:12, Peter Eisentraut wrote:
On 5/5/17 07:13, Petr Jelinek wrote:
Yes that's different thing that we've been discussing a bit in snapbuild
woes thread.the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)Thanks, so the patch works.
Committed, with s/xact_started/started_tx/, to match nearby code.
Thanks!
--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On Tue, May 9, 2017 at 1:26 AM, Petr Jelinek
<petr.jelinek@2ndquadrant.com> wrote:
On 08/05/17 17:52, Masahiko Sawada wrote:
On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek
<petr.jelinek@2ndquadrant.com> wrote:On 03/05/17 13:23, Erik Rijkers wrote:
On 2017-05-03 08:17, Petr Jelinek wrote:
On 02/05/17 20:43, Robert Haas wrote:
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut
code path that calls CommitTransactionCommand() should have one, no?
Is there anything left to be committed here?
Afaics the fix was not committed. Peter wanted more comprehensive fix
which didn't happen. I think something like attached should do the job.I'm running my pgbench-over-logical-replication test in chunk of 15
minutes, wth different pgbench -c (num clients) and -s (scale) values.With this patch (and nothing else) on top of master (8f8b9be51fd7 to be
precise):fix-statistics-reporting-in-logical-replication-work.patch
logical replication is still often failing (as expected, I suppose; it
seems because of "inital snapshot too large") but indeed I do not seeYes that's different thing that we've been discussing a bit in snapbuild
woes thread.the 'TRAP: FailedAssertion in pgstat.c' anymore.
(If there is any other configuration of patches worth testing please let
me know)Thanks, so the patch works.
I think that we should commit the local transaction that did initial
data copy, and then report stat as well. Currently table sync worker
doesn't commit the local transaction in LogicalRepSyncTableStart
(maybe until apply commit record?) if its status is changed to
SUBREL_STATE_CATCHUP. That's why the table sync worker issues
assertion failure.That would fix the assert as well yes, but it would also mean that if
the worker crashed between the initial copy and the end of catchup there
would be no way to restart it without manual intervention from user
since the synchronization position would be lost. Hence the fix I
proposed which does it differently and has the whole sync in a single
transaction.
I understood that the data synchronization even including apply
logical record after changed to SUBREL_STATE_CATCHUP should be done in
a single transaction. Thank you for explanation.
Regards,
--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers