Standby server crashes in master and REL9_5_STABLE branches

Started by Nonameover 10 years ago4 messages
#1Noname
oonishitk@nttdata.co.jp

Hi,

Today, I found that standby server crashes with latest code of master branch and REL9_5_STABLE branch.
This happens not always but sometimes after doing pg_start_backup and pg_stop_backup for master server.

The below error messages were shown in standby server log:

FATAL: could not access status of transaction 9009
DETAIL: Could not read from file "pg_commit_ts/0000" at offset 90112: Success.
CONTEXT: xlog redo Transaction/COMMIT: 2015-09-30 15:52:41.924141+09
LOG: startup process (PID 23199) exited with exit code 1
LOG: terminating any other active server processes

Before this FATAL, there were some INFO but annoying messages:

LOG: file "pg_commit_ts/0000" doesn't exist, reading as zeroes
CONTEXT: xlog redo Transaction/COMMIT: 2015-09-30 15:47:14.747566+09; inval msgs: catcache 49 catcache 49 catcache 49 catcache 49

I cannot explain why but this crash seems to disappear if I moved HEAD to before commit 6b61955135e94b39d85571fdbb0c5a749af767f1.

Regards,

========
Takashi Ohnishi
oonishitk@nttdata.co.jp

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

#2Michael Paquier
michael.paquier@gmail.com
In reply to: Noname (#1)
Re: Standby server crashes in master and REL9_5_STABLE branches

On Wed, Sep 30, 2015 at 4:15 PM, <oonishitk@nttdata.co.jp> wrote:

FATAL: could not access status of transaction 9009
DETAIL: Could not read from file "pg_commit_ts/0000" at offset 90112:
Success.
CONTEXT: xlog redo Transaction/COMMIT: 2015-09-30 15:52:41.924141+09
LOG: startup process (PID 23199) exited with exit code 1
LOG: terminating any other active server processes

I cannot explain why but this crash seems to disappear if I moved HEAD to
before commit 6b61955135e94b39d85571fdbb0c5a749af767f1.

Even if I have the feeling that Alvaro, Petr and/or Fujii-san already know
what is going on, do you have a backtrace at hand?
--
Michael

#3Noname
oonishitk@nttdata.co.jp
In reply to: Michael Paquier (#2)
Re: Standby server crashes in master and REL9_5_STABLE branches

No, I do not have a backtrace.

I 'm sorry about using misleading word ‘crash’.
The standby server did not cause process crash.
It exited abnormally.

Regards,

======
Takashi Ohnishi
oonishitk@nttdata.co.jp

From: Michael Paquier [mailto:michael.paquier@gmail.com]
Sent: Wednesday, September 30, 2015 4:30 PM
To: SPS 大西 高史(三技術) <oonishitk@nttdata.co.jp>
Cc: PostgreSQL mailing lists <pgsql-hackers@postgresql.org>; Masao Fujii <masao.fujii@gmail.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Petr Jelinek <petr@2ndquadrant.com>
Subject: Re: [HACKERS] Standby server crashes in master and REL9_5_STABLE branches

On Wed, Sep 30, 2015 at 4:15 PM, <oonishitk@nttdata.co.jp<mailto:oonishitk@nttdata.co.jp>> wrote:
FATAL: could not access status of transaction 9009
DETAIL: Could not read from file "pg_commit_ts/0000" at offset 90112: Success.
CONTEXT: xlog redo Transaction/COMMIT: 2015-09-30 15:52:41.924141+09
LOG: startup process (PID 23199) exited with exit code 1
LOG: terminating any other active server processes

I cannot explain why but this crash seems to disappear if I moved HEAD to before commit 6b61955135e94b39d85571fdbb0c5a749af767f1.

Even if I have the feeling that Alvaro, Petr and/or Fujii-san already know what is going on, do you have a backtrace at hand?
--
Michael

#4Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#2)
Re: Standby server crashes in master and REL9_5_STABLE branches

Michael Paquier wrote:

On Wed, Sep 30, 2015 at 4:15 PM, <oonishitk@nttdata.co.jp> wrote:

FATAL: could not access status of transaction 9009
DETAIL: Could not read from file "pg_commit_ts/0000" at offset 90112:
Success.
CONTEXT: xlog redo Transaction/COMMIT: 2015-09-30 15:52:41.924141+09
LOG: startup process (PID 23199) exited with exit code 1
LOG: terminating any other active server processes

I cannot explain why but this crash seems to disappear if I moved HEAD to
before commit 6b61955135e94b39d85571fdbb0c5a749af767f1.

Sigh. Will fix.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers