Please help! Server process was terminated by signal 11: Segmentation fault
Hello Everyone, On 2015/12/30 17:51:01, Transaction was failed on
our system. I checked the pg_log, it states as below: 2015-12-30
17:51:01 CST [11260]: [42-1] LOG: 00000: server process (PID 14272) was
terminated by signal 11: Segmentation fault <part of the pg_log>================================================ 2015-12-30 17:50:52 CST [18954]: [5165-1] LOG: 00000: statement: select
se_addlog('1','99','20151230161941915390111750','O','0000','');2015-12-30 17:50:52 CST [18954]: [5166-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5167-1] LOG: 00000: duration: 0.951 ms2015-12-30 17:50:52 CST [18954]: [5168-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5169-1] LOG: 00000: statement: unlisten *2015-12-30 17:50:52 CST [18954]: [5170-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5171-1] LOG: 00000: duration: 0.118 ms2015-12-30 17:50:52 CST [18954]: [5172-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5173-1] LOG: 00000: statement: select 12015-12-30 17:50:52 CST [18954]: [5174-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5175-1] LOG: 00000: duration: 0.160 ms2015-12-30 17:50:52 CST [18954]: [5176-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5179-1] LOG: 00000: duration: 0.738 ms2015-12-30 17:50:52 CST [18954]: [5180-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5181-1] LOG: 00000: statement: unlisten *2015-12-30 17:50:52 CST [18954]: [5182-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5183-1] LOG: 00000: duration: 0.112 ms2015-12-30 17:50:52 CST [18954]: [5184-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5185-1] LOG: 00000: statement: select 12015-12-30 17:50:52 CST [18954]: [5186-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5187-1] LOG: 00000: duration: 0.159 ms2015-12-30 17:50:52 CST [18954]: [5188-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:50:52 CST [18954]: [5190-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5191-1] LOG: 00000: duration: 0.832 ms2015-12-30 17:50:52 CST [18954]: [5192-1] LOCATION: exec_simple_query, postgres.c:13642015-12-30 17:50:52 CST [18954]: [5193-1] LOG: 00000: statement: unlisten *2015-12-30 17:50:52 CST [18954]: [5194-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:50:52 CST [18954]: [5195-1] LOG: 00000: duration: 0.111 ms2015-12-30 17:50:52 CST [18954]: [5196-1]
LOCATION: exec_simple_query,
postgres.c:1364★2015-12-30 17:51:02 CST [14276]: [1-1] LOG: 00000: database system was interrupted; last
known up at 2015-12-30 17:50:21 CST★★2015-12-30 17:51:02 CST [14276]: [31-1] LOG: 00000: last completed transaction was at log
time 2015-12-30 17:50:52.793516+08★2015-12-30 17:51:01 CST [14270]: [1-1] LOG: 00000: connection received: host=[local]2015-12-30 17:51:01 CST [14270]: [2-1]
LOCATION: BackendInitialize,
postmaster.c:40562015-12-30 17:51:01 CST [14270]: [3-1] LOG: 00000: connection authorized:
user=enterprisedb database=edb2015-12-30 17:51:01 CST [14270]: [4-1]
LOCATION: PerformAuthentication,
postinit.c:2652015-12-30 17:51:01 CST [14271]: [1-1] LOG: 00000: connection received: host=[local]2015-12-30 17:51:01 CST [14271]: [2-1]
LOCATION: BackendInitialize,
postmaster.c:40562015-12-30 17:51:01 CST [14271]: [3-1] LOG: 00000: connection authorized:
user=enterprisedb database=edb2015-12-30 17:51:01 CST [14271]: [4-1]
LOCATION: PerformAuthentication,
postinit.c:2652015-12-30 17:51:01 CST [14270]: [5-1] LOG: 00000: statement: select now();2015-12-30 17:51:01 CST [14270]: [6-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:51:01 CST [14272]: [1-1] LOG: 00000: connection received: host=[local]2015-12-30 17:51:01 CST [14272]: [2-1]
LOCATION: BackendInitialize,
postmaster.c:40562015-12-30 17:51:01 CST [14270]: [7-1] LOG: 00000: duration: 1.523 ms2015-12-30 17:51:01 CST [14270]: [8-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:51:01 CST [14270]: [9-1] LOG: 00000: statement: select
current_timestamp,pid,current_timestamp - query_start as
runtime,datname,usename,client_addr,query 2015-12-30 17:51:01 CST [14270]: [10-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:51:01 CST [14272]: [3-1] LOG: 00000: connection authorized:
user=enterprisedb database=edb2015-12-30 17:51:01 CST [14272]: [4-1]
LOCATION: PerformAuthentication,
postinit.c:2652015-12-30 17:51:01 CST [14271]: [5-1] LOG: 00000: statement: select
CURRENT_TIMESTAMP(0), client_addr, count(client_addr) from pg_stat_activity
group by client_addr order by client_addr;2015-12-30 17:51:01 CST [14271]: [6-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:51:01 CST [14272]: [5-1] LOG: 00000: statement: select now();2015-12-30 17:51:01 CST [14272]: [6-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:51:01 CST [14272]: [7-1] LOG: 00000: duration: 0.997 ms2015-12-30 17:51:01 CST [14272]: [8-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:51:01 CST [14272]: [9-1] LOG: 00000: statement: SELECT bl.pid AS blocked_pid,2015-12-30 17:51:01 CST [14272]: [10-1]
LOCATION: exec_simple_query,
postgres.c:10822015-12-30 17:51:01 CST [14270]: [11-1] LOG: 00000: duration: 5.044 ms2015-12-30 17:51:01 CST [14270]: [12-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:51:01 CST [14270]: [13-1] LOG: 00000: disconnection: session time:
0:00:00.015 user=enterprisedb database=edb host=[local]2015-12-30 17:51:01 CST [14270]: [14-1]
LOCATION: log_disconnections,
postgres.c:59932015-12-30 17:51:01 CST [14271]: [7-1] LOG: 00000: duration: 7.072 ms2015-12-30 17:51:01 CST [14271]: [8-1]
LOCATION: exec_simple_query,
postgres.c:13642015-12-30 17:51:01 CST [14271]: [9-1] LOG: 00000: disconnection: session time:
0:00:00.015 user=enterprisedb database=edb host=[local]2015-12-30 17:51:01 CST [14271]: [10-1]
LOCATION: log_disconnections,
postgres.c:5993★2015-12-30 17:51:01 CST [11260]: [42-1] LOG: 00000: server process (PID 14272) was
terminated by signal 11: Segmentation fault★★2015-12-30 17:51:01 CST [11260]: [43-1] DETAIL: Failed process was running: SELECT
bl.pid AS blocked_pid,★2015-12-30 17:51:01 CST [11260]: [44-1]
LOCATION: LogChildExit,
postmaster.c:33622015-12-30 17:51:01 CST [11260]: [45-1] LOG: 00000: terminating any other active server
processes2015-12-30 17:51:01 CST [11260]: [46-1]
LOCATION: HandleChildCrash,
postmaster.c:30832015-12-30 17:51:01 CST [18954]: [5197-1]
WARNING: 57P02: terminating connection
because of crash of another server process2015-12-30 17:51:01 CST [18954]: [5198-1] DETAIL: The postmaster has commanded this server
process to roll back the current transaction and exit, because another server
process exited abnormally and possibly corrupted shared memory.2015-12-30 17:51:01 CST [18954]: [5199-1]
HINT: In a moment you should be able to
reconnect to the database and repeat your command.2015-12-30 17:51:01 CST [18954]: [5200-1]
LOCATION: quickdie, postgres.c:38552015-12-30 17:51:01 CST [18953]: [4969-1]
WARNING: 57P02: terminating connection
because of crash of another server process2015-12-30 17:51:01 CST [18953]: [4970-1]
DETAIL: The postmaster has commanded
this server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared memory.2015-12-30 17:51:01 CST [18953]: [4971-1]
HINT: In a moment you should be able to
reconnect to the database and repeat your command.2015-12-30 17:51:01 CST [18953]: [4972-1]
LOCATION: quickdie, postgres.c:3855 ================================================ The server log states:Dec 30
17:51:01 egtcdb1 kernel: edb-postgres[14272]: segfault at 30 ip 00000000006a1f5a
sp 00007fff70c5e2e0 error 4 in edb-postgres[400000+5e8000] Server
information,egtcdb1
(Linux release 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41 EST 2013) (0)[enterprisedb@egtcdb1
~]$ psqlpsql.bin
(9.4.1.3)Type
"help" for help.edb=# select
version ; LOG:
statement: select version ;LOG:
duration: 1.603 ms
version
-------------------------------------------------------------------------------------------------------------------EnterpriseDB
9.4.1.3 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red
Hat 4.1.2-55), 64-bit(1 row) [enterprisedb@egtcdb1
~]$ free -tm
total
used free
shared buffers cachedMem:
15888 12506
3381
0 502
11204-/+
buffers/cache:
799 15088Swap:
4015
36 3979Total:
19904
12543 7361 Cpu 4
core, memory 16GB Postgresql.conf
information,shared_buffers
= 4GBtemp_buffers
= 8MBwork_mem
= 56696kBmaintenance_work_mem
= 256MBeffective_cache_size
= 5324784kB On my
case, Please give me the instruction in steps on how to figure out the reason
why did this “signal 11” happen and how to solve “signal 11: Segmentation fault”
issue. Thank you
in advance.John
On 1/4/2016 6:07 PM, John Hong wrote:
EnterpriseDB 9.4.1.3 on x86_64-unknown-linux-gnu, compiled by gcc
(GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit
you probably should contact EnterpriseDB technical support. This email
list is for the community open source version of PostgreSQL.
--
john r pierce, recycling bits in santa cruz
On Tue, Jan 5, 2016 at 12:36 PM, John R Pierce <pierce@hogranch.com> wrote:
On 1/4/2016 6:07 PM, John Hong wrote:
EnterpriseDB 9.4.1.3 on x86_64-unknown-linux-gnu, compiled by gcc (GCC)
4.1.2 20080704 (Red Hat 4.1.2-55), 64-bityou probably should contact EnterpriseDB technical support. This email
list is for the community open source version of PostgreSQL.
If you actually come up with a self-contained test case that fails
with the community version, then here would be fine.
--
Michael
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On 01/04/2016 06:07 PM, John Hong wrote:
Hello Everyone,
On 2015/12/30 17:51:01, Transaction was failed on our system.
I checked the pg_log, it states as below:
2015-12-30 17:51:01 CST [11260]: [42-1] LOG: 00000: server process (PID
14272) was terminated by signal 11: Segmentation fault
On my case, Please give me the instruction in steps on how to figure out
thereason why did this “signal 11” happen and how to solve “signal 11:
Segmentation fault” issue.
The part that confuses me is:
"
2015-12-30 17:50:52 CST [18954]: [5195-1] LOG: 00000: duration: 0.111 ms
2015-12-30 17:50:52 CST [18954]: [5196-1] LOCATION: exec_simple_query,
postgres.c:1364
★2015-12-30 17:51:02 CST [14276]: [1-1] LOG: 00000: database system was
interrupted; last known up at 2015-12-30 17:50:21 CST★
★2015-12-30 17:51:02 CST [14276]: [31-1] LOG: 00000: last completed
transaction was at log time 2015-12-30 17:50:52.793516+08★
2015-12-30 17:51:01 CST [14270]: [1-1] LOG: 00000: connection received:
host=[local]
"
Is the above a cut and paste error or did the server really go backwards
in time?
At any rate this:
"
2015-12-30 17:51:01 CST [14270]: [9-1] LOG: 00000: statement: select
current_timestamp,pid,current_timestamp - query_start as
runtime,datname,usename,client_addr,query
2015-12-30 17:51:01 CST [14270]: [10-1] LOCATION: exec_simple_query,
postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [3-1] LOG: 00000: connection
authorized: user=enterprisedb database=edb
2015-12-30 17:51:01 CST [14272]: [4-1] LOCATION: PerformAuthentication,
postinit.c:265
2015-12-30 17:51:01 CST [14271]: [5-1] LOG: 00000: statement: select
CURRENT_TIMESTAMP(0), client_addr, count(client_addr) from
pg_stat_activity group by client_addr order by client_addr;
2015-12-30 17:51:01 CST [14271]: [6-1] LOCATION: exec_simple_query,
postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [5-1] LOG: 00000: statement: select now();
2015-12-30 17:51:01 CST [14272]: [6-1] LOCATION: exec_simple_query,
postgres.c:1082
2015-12-30 17:51:01 CST [14272]: [7-1] LOG: 00000: duration: 0.997 ms
2015-12-30 17:51:01 CST [14272]: [8-1] LOCATION: exec_simple_query,
postgres.c:1364
2015-12-30 17:51:01 CST [14272]: [9-1] LOG: 00000: statement: SELECT
bl.pid AS blocked_pid,
"
which includes the statement that was at fault:
"★2015-12-30 17:51:01 CST [11260]: [42-1] LOG: 00000: server process
(PID 14272) was terminated by signal 11: Segmentation fault★
★2015-12-30 17:51:01 CST [11260]: [43-1] DETAIL: Failed process was
running: SELECT bl.pid AS blocked_pid,★
"
looks like something coming from a monitoring/logging process, if that
helps.
Thank you in advance.
John
--
Adrian Klaver
adrian.klaver@aklaver.com
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general