BDR workers exiting?

On 10/12/15 9:37 AM, Steve Pribyl wrote:

I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is something going bad.

2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr (6204748238611542317,1,16494,): apply"
2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,""
2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,""

Looks like something's going bad, but you need to ask on the BDR mailing
list.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 10/12/15 10:14 AM, Jim Nasby wrote:

On 10/12/15 9:37 AM, Steve Pribyl wrote:

I am loading up a 60G database into BDR database and these "ERRORS"
are in my logs. Is not normal behavior or is something going bad.

2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12
09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr
(6204748238611542317,1,16494,): apply"
2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12
08:12:14 CDT,,0,LOG,00000,"worker process: bdr
(6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID
30371) exited with exit code 1",,,,,,,,,""
2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12
08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr
(6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,""

Looks like something's going bad, but you need to ask on the BDR mailing
list.

Nevermind, just discovered there is no separate list. Sorry for the noise.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote:

I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is something going bad.

2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr (6204748238611542317,1,16494,): apply"
2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,""
2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,""

There'll possibly be an error message on the other node about ending the
connection.

Do you use SSL? If so, try disabling renegotiation.

Regards,

Andres

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Yup, there is a disconnect on other side.

This disconnect is preceded by this.
ERROR,XX000,"invalid memory alloc request size 1073741824",,,,,"slot ""bdr_16494_6204748238611542317_1_16494__"", output plugin ""bdr"", in the change callback, associated LSN 2/FD250E48",,,,"bdr (6204748238611542317,1,16494,):receive"

Steve Pribyl
Sr. Systems Engineer
steve.pribyl@akunacapital.com
Desk: 312-994-4646

________________________________________
From: Andres Freund <andres@anarazel.de>
Sent: Monday, October 12, 2015 11:08 AM
To: Steve Pribyl
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] BDR workers exiting?

On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote:

I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is something going bad.

2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr (6204748238611542317,1,16494,): apply"
2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,""
2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,""

There'll possibly be an error message on the other node about ending the
connection.

Do you use SSL? If so, try disabling renegotiation.

Regards,

Andres
________________________________
[http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com&gt;
p: +1 312 994 4646 | m: | f: +1 312 750 1667 | Steve.Pribyl@akunacapital.com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwise protected from disclosure. This information is intended for the use of the addressee only and is not offered as investment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recorded and stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, the contents of this electronic message is strictly prohibited. If you have received this communication in error, please notify us by telephone at (312)994-4640 and destroy the original message.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

The process used to created this

Start with clean db
Create host A database with bdr
Join host B with dbr
Load database using psql < file.sql

I was able to get it work if I do the following.
Start with clean db
Create host A database
Load data on host A
Join host A to bdr.
Join host b to bdr.

Glad to have a work around but would like to get to understand the failure.

Steve Pribyl

________________________________________
From: Steve Pribyl
Sent: Monday, October 12, 2015 11:19 AM
To: Andres Freund
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] BDR workers exiting?

Yup, there is a disconnect on other side.

This disconnect is preceded by this.
ERROR,XX000,"invalid memory alloc request size 1073741824",,,,,"slot ""bdr_16494_6204748238611542317_1_16494__"", output plugin ""bdr"", in the change callback, associated LSN 2/FD250E48",,,,"bdr (6204748238611542317,1,16494,):receive"

Steve Pribyl

________________________________________
From: Andres Freund <andres@anarazel.de>
Sent: Monday, October 12, 2015 11:08 AM
To: Steve Pribyl
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] BDR workers exiting?

On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote:

I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is something going bad.

2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr (6204748238611542317,1,16494,): apply"
2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,""
2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,""

There'll possibly be an error message on the other node about ending the
connection.

Do you use SSL? If so, try disabling renegotiation.

Regards,

Andres
________________________________
[http://www.akunacapital.com/images/akuna.png]
Steve Pribyl | Senior Systems Engineer
Akuna Capital LLC
36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com&gt;
p: +1 312 994 4646 | m: | f: +1 312 750 1667 | Steve.Pribyl@akunacapital.com

Please consider the environment, before printing this email.

This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwise protected from disclosure. This information is intended for the use of the addressee only and is not offered as investment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recorded and stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, the contents of this electronic message is strictly prohibited. If you have received this communication in error, please notify us by telephone at (312)994-4640 and destroy the original message.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

BDR is currently memory-limited for extremely large transactions. At a
guess, I'd say one of your big tables is large enough that the logical
decoding facility BDR uses can't keep track of the transaction
properly.

There's no hard limit, it depends on details of the transaction and a
number of other variables, but "many tens or hundreds of GB" is
generally too much.

If I was to load such a big DB, I'd probably do it with ETL tools that
could split up the load and do it progressively.

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general