BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________
From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

On Wed, Jan 15, 2020 at 04:02:38AM +0000, Pendekar Dikala Senja wrote:

Can you help us to find out the solution and configuration?

A more detailed backtrace would be nice. Here are some guidelines:
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
--
Michael

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <
pendekar.senja@outlook.com> wrote:

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6

to 12.1, but if there is no partition table the replication works so well.

As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version:

core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch

Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________ From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16205: background worker "logical replication worker"

(PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL

11.6

to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already

replicated

around more than 2 hours, server can't access and recovery mode. please

help

to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:

/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

*How to get stack trace:*
After crash, you can see core.PID file in your data directory. Attach core
file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________ From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.
(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________ From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <
pendekar.senja@outlook.com> wrote:

following your intruction. please find the information from my server
below after running stact trace. I can't capture the PID because PID is
always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating
the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

(gdb) set pagination off

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan 2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_commit_ts
-rw------- 1 postgres postgres 3 Jan 2 16:34 PG_VERSION
-rw------- 1 postgres postgres 88 Jan 2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan 2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan 3 18:29 pg_replslot
-rw------- 1 postgres postgres 127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres 47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres 52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres 93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
5432001 169312256
ready

________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com&lt;http://www.enterprisedb.com/&gt;

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________ From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

please find below :

sudo gdb -p 15803
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 15803
Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/b1/7c21299099640a6d863e423d99265824e7bb16.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicui18n.so.55...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicuuc.so.55...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.30...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicudata.so.55...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.4...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_stat_statements.so...(no debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_partman_bgw.so...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.
done.
0x00007f5ce4bf85b3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.

________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 7:44 PM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan 2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_commit_ts
-rw------- 1 postgres postgres 3 Jan 2 16:34 PG_VERSION
-rw------- 1 postgres postgres 88 Jan 2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan 2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan 2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan 3 18:29 pg_replslot
-rw------- 1 postgres postgres 127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres 47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres 52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres 93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
5432001 169312256
ready

________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.

sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com&lt;http://www.enterprisedb.com/&gt;

(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded. Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)

thanks

________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.

Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:

Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________ From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:

I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.

Could you provide a backtrace? Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
/messages/by-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

We've been encountering the same bug; here's a stack trace from the crashed process:

(gdb) bt
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:427
#1 0x00005582bda1fb57 in memcpy (__len=425318492, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#2 tts_virtual_materialize (slot=0x5582bebca418) at ./build/../src/backend/executor/execTuples.c:235
#3 0x00005582bda21179 in ExecFetchSlotHeapTuple (slot=0x5582bebca418, materialize=<optimized out>, shouldFree=0x7ffeccfccc37) at ./build/../src/backend/executor/execTuples.c:1624
#4 0x00005582bd8b4e9c in heapam_tuple_update (relation=0x7f0bf1a92430, otid=0x5582bebca6a8, slot=0x5582bebca418, cid=943, snapshot=<optimized out>, crosscheck=0x0, wait=true,
tmfd=0x7ffeccfcccc0, lockmode=0x7ffeccfcccbc, update_indexes=0x7ffeccfccd37) at ./build/../src/backend/access/heap/heapam_handler.c:325
#5 0x00005582bd8e4306 in table_tuple_update (update_indexes=0x7ffeccfccd37, lockmode=0x7ffeccfcccbc, tmfd=0x7ffeccfcccc0, wait=true, crosscheck=0x0, snapshot=0x0,
cid=<optimized out>, slot=0x5582bebca418, otid=0x5582bebca6a8, rel=0x7f0bf1a92430) at ./build/../src/include/access/tableam.h:1261
#6 simple_table_tuple_update (rel=0x7f0bf1a92430, otid=otid@entry=0x5582bebca6a8, slot=slot@entry=0x5582bebca418, snapshot=0x0, update_indexes=update_indexes@entry=0x7ffeccfccd37)
at ./build/../src/backend/access/table/tableam.c:325
#7 0x00005582bda1d462 in ExecSimpleRelationUpdate (estate=estate@entry=0x5582bebd1970, epqstate=epqstate@entry=0x7ffeccfcce10, searchslot=searchslot@entry=0x5582bebca678,
slot=slot@entry=0x5582bebca418) at ./build/../src/backend/executor/execReplication.c:493
#8 0x00005582bdb1a188 in apply_handle_update (s=s@entry=0x7ffeccfd4420) at ./build/../src/backend/replication/logical/worker.c:750
#9 0x00005582bdb1affe in apply_dispatch (s=0x7ffeccfd4420) at ./build/../src/backend/replication/logical/worker.c:968
#10 LogicalRepApplyLoop (last_received=8839008698464) at ./build/../src/backend/replication/logical/worker.c:1176
#11 ApplyWorkerMain (main_arg=<optimized out>) at ./build/../src/backend/replication/logical/worker.c:1734
#12 0x00005582bdae972e in StartBackgroundWorker () at ./build/../src/backend/postmaster/bgworker.c:834
#13 0x00005582bdaf579c in do_start_bgworker (rw=0x5582beafcca0) at ./build/../src/backend/postmaster/postmaster.c:5770
#14 maybe_start_bgworkers () at ./build/../src/backend/postmaster/postmaster.c:5996
#15 0x00005582bdaf6235 in sigusr1_handler (postgres_signal_arg=<optimized out>) at ./build/../src/backend/postmaster/postmaster.c:5167
#16 <signal handler called>
#17 0x00007f1013bfaff7 in __GI___select (nfds=nfds@entry=8, readfds=readfds@entry=0x7ffeccfd5470, writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0,
timeout=timeout@entry=0x7ffeccfd53d0) at ../sysdeps/unix/sysv/linux/select.c:41
#18 0x00005582bdaf67db in ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1668
#19 0x00005582bdaf8513 in PostmasterMain (argc=5, argv=0x5582beaaa090) at ./build/../src/backend/postmaster/postmaster.c:1377
#20 0x00005582bd86ecf6 in main (argc=5, argv=0x5582beaaa090) at ./build/../src/backend/main/main.c:228
(gdb)
--
-- Christophe Pettus
xof@thebuild.com

On Thu, Jan 16, 2020 at 10:46:55AM -0800, Christophe Pettus wrote:

We've been encountering the same bug; here's a stack trace from the
crashed process:

[stack trace]

I cannot say if Pendekar's bug is the same issue, but yours looks very
familiar to that:
/messages/by-id/16129-a0c0f48e71741e5f@postgresql.org

And this has already been fixed, in time for 12.2.
--
Michael