stuck spinlock detected at dynahash.c

Started by Timonover 15 years ago5 messagesgeneral

timosha@gmail.com

over 15 years ago

today I found in logs:

WARNING: pgstat wait timeout
WARNING: pgstat wait timeout
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
...
LOG: server process (PID 8078) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process
exited a
....
LOG: archiver process (PID 6277) exited with exit code 1
FATAL: the database system is in recovery mode
....
LOG: all server processes terminated; reinitializing
.....
LOG: database system was interrupted; last known up at 2010-10-06 15:03:32 MSD
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at B7/3FF16C28
LOG: record with zero length at B7/4B810F58
LOG: redo done at B7/4B810F28
LOG: last completed transaction was at log time 2010-10-06 15:06:44.254586+04
LOG: autovacuum launcher started
LOG: database system is ready to accept connections

what was that?
I google only http://archives.postgresql.org/pgsql-general/2009-11/msg00871.php

postgresql-8.4.4-1.fc13.x86_64

--
All bugs reserved

Alvaro Herrera

alvherre@2ndquadrant.com

over 15 years ago

In reply to: Timon (#1)

Re: stuck spinlock detected at dynahash.c

Excerpts from Timon's message of mié oct 06 07:35:44 -0400 2010:

today I found in logs:

WARNING: pgstat wait timeout
WARNING: pgstat wait timeout
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981

What platform is this? Are you running on some sort of virtualized
system?

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Timon

timosha@gmail.com

over 15 years ago

In reply to: Alvaro Herrera (#2)

Re: stuck spinlock detected at dynahash.c

no, simple dedicated server with Linux Fedora 13 x86_64
kernel-2.6.33.6-147.2.4.fc13.x86_64

2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>:

Excerpts from Timon's message of mié oct 06 07:35:44 -0400 2010:

today I found in logs:

WARNING: pgstat wait timeout
WARNING: pgstat wait timeout
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981

What platform is this? Are you running on some sort of virtualized
system?

--
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

--
All bugs reserved

Tom Lane

tgl@sss.pgh.pa.us

over 15 years ago

In reply to: Timon (#3)

Re: stuck spinlock detected at dynahash.c

Timon <timosha@gmail.com> writes:

2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>:

WARNING: �pgstat wait timeout
WARNING: �pgstat wait timeout
PANIC: �stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
PANIC: �stuck spinlock (0x7fd877761af8) detected at dynahash.c:981

What platform is this? �Are you running on some sort of virtualized
system?

no, simple dedicated server with Linux Fedora 13 x86_64
kernel-2.6.33.6-147.2.4.fc13.x86_64

There's not a lot we can do with only this much information. I'd
suggest running the postmaster with core dumps enabled (ulimit -c
unlimited) so that if it happens again, the PANICs will result in
core dumps that you can get stack traces from.

regards, tom lane

Timon

timosha@gmail.com

over 15 years ago

In reply to: Timon (#1)

Fwd: stuck spinlock detected at dynahash.c

---------- Forwarded message ----------
From: Timon <timosha@gmail.com>
Date: 2010/10/8
Subject: Re: [GENERAL] stuck spinlock detected at dynahash.c
To: Tom Lane <tgl@sss.pgh.pa.us>

I have this (thanks abrtd):
-------------------------------
Crash ID: 26:a6235a787c4e9c6c5ab4f6eadd2168371b4e0fe3
Last crash: Wed 06 Oct 2010 03:13:42 PM MSD
Analyzer: CCpp
Component: postgresql
Package: postgresql-server-8.4.4-1.fc13
Command: 'postgres: autovacuum worker process ' '' '' '' '' ''
Executable: /usr/bin/postgres
System: Fedora release 13 (Goddard), kernel
2.6.33.6-147.2.4.fc13.x86_64
Rating: 0
Coredump file: /var/spool/abrt/ccpp-1286363576-2966/coredump
Reason: Process /usr/bin/postgres was killed by signal 6 (SIGABRT)
Hostname: xxxxxxxxx

Backtrace:
BFD: Warning: /var/spool/abrt/ccpp-1286363576-2966/coredump is
truncated: expected core file size >= 8587022336, found: 65536.
warning: core file may not match specified executable file.
[New Thread 2966]
Cannot access memory at address 0x33d3820108
Cannot access memory at address 0x33d3820108
Cannot access memory at address 0x33d3820108
Cannot access memory at address 0x33d3820108
Cannot access memory at address 0x33d3820108
Failed to read a valid object file image from memory.
Core was generated by `postgres: autovacuum worker process '.
Program terminated with signal 6, Aborted.
#0 0x00000033d3a329a5 in ?? ()

Thread 1 (Thread 2966):
#0 0x00000033d3a329a5 in ?? ()
No symbol table info available.
Cannot access memory at address 0x7fffaebbae78
Cannot access memory at address 0x33d3820108

From To Syms Read Shared Object Library

0x000000306fe00af0 0x000000306fe18904 Yes /lib64/ld-linux-x86-64.so.2
No symbol "__abort_msg" in current context.
No symbol "__glib_assert_msg" in current context.
rax 0x0 0
rbx 0xac0840 11274304
rcx 0xffffffffffffffff -1
rdx 0x6 6
rsi 0xb96 2966
rdi 0xb96 2966
rbp 0x16 0x16
rsp 0x7fffaebbae78 0x7fffaebbae78
r8 0x7fd87ee427c0 140567818545088
r9 0x616e796420746120 7020682340065370400
r10 0x8 8
r11 0x206 518
r12 0x0 0
r13 0x282d870 42129520
r14 0x282d870 42129520
r15 0x7da50b 8234251
rip 0x33d3a329a5 0x33d3a329a5
eflags 0x206 [ PF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
No function contains program counter for selected frame.
-------------------------------

maybe this can help a little

2010/10/7 Tom Lane <tgl@sss.pgh.pa.us>:

Timon <timosha@gmail.com> writes:

2010/10/6 Alvaro Herrera <alvherre@commandprompt.com>:

WARNING: pgstat wait timeout
WARNING: pgstat wait timeout
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981
PANIC: stuck spinlock (0x7fd877761af8) detected at dynahash.c:981

What platform is this? Are you running on some sort of virtualized
system?

no, simple dedicated server with Linux Fedora 13 x86_64
kernel-2.6.33.6-147.2.4.fc13.x86_64

There's not a lot we can do with only this much information. I'd
suggest running the postmaster with core dumps enabled (ulimit -c
unlimited) so that if it happens again, the PANICs will result in
core dumps that you can get stack traces from.

regards, tom lane

--
All bugs reserved

Import Notes

Reply to msg id not found: AANLkTinoO6JYCuV2VaW6kak+mx7nQpbh=3k_ZFqn1i+G@mail.gmail.com