PANIC serves too many masters

Started by Andres Freundabout 3 years ago13 messageshackers
Jump to latest
#1Andres Freund
andres@anarazel.de

Hi,

Right now we use PANIC for very different kinds of errors.

Sometimes for errors that are persistent, where crash-restarting and trying
again won't help:
ereport(PANIC,
(errmsg("could not locate a valid checkpoint record")));
or
ereport(PANIC,
(errmsg("online backup was canceled, recovery cannot continue")));

Sometimes for errors that could be transient, e.g. when running out of space
while trying to write out WAL:
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not write to file \"%s\": %m", tmppath)));
(the ERROR is often promoted to a PANIC due to critical sections).
or
ereport(PANIC,
(errcode_for_file_access(),
errmsg("could not write to log file \"%s\" at offset %u, length %zu: %m",
xlogfname, startoffset, nleft)));

Sometimes for "should never happen" checks that are important enough to check
in production builds:
elog(PANIC, "stuck spinlock detected at %s, %s:%d",
or
elog(PANIC, "failed to re-find shared proclock object");

I have two main issues with this:

1) If core dumps are allowed, we trigger core dumps for all of these. That's
good for "should never happen" type of errors like a stuck spinlock. But it
makes no sense for things like the on-disk state being wrong at startup, or
running out of space while writing WAL - if anything, it might make that
worse!

It's very useful to be able to collect dumps for crashes in production, but
it's not useful to generate thousands of identical cores because crash
recovery fails with out-of-space and we retry over and over.

2) For errors where crash-restarting won't fix anything, using PANIC doesn't
allow postmaster to distinguish between an error that should lead
postmaster to exit itself (after killing other processes, obviously) and
the normal crash restart cycle.

I've been trying to do some fleet wide analyses of the causes of crashes, but
having core dumps for lots of stuff that aren't crashes, often repeated many
times, makes that much harder. Filtering out abort()s and just looking at
segfaults filters out far too much.

I don't quite know what we should do. But the current situation decidedly
doesn't seem great.

Maybe we could have:
- PANIC_BUG - triggers abort() followed by a crash restart cycle
to be used for things like a stuck spinlock
- PANIC_RETRY - causes a crash restart cycle, no core dump
to be used for things like ENOSPC during WAL writes
- PANIC_EXIT - causes postmaster to exit(1)
to be used for things where retrying won't help, like
"requested recovery stop point is before consistent recovery point"

One could argue that some of the PANICs that want to just shut down the server
should instead be FATALs, with knowledge in postmaster about which/when such
errors should trigger exiting. We do have something like this for the startup
process, but only when errors happen "early enough", and without being able to
distinguish between "retryable" and "should exit" type errors. But ISTM that
that requires adding more and more knowledge to postmaster.c, instead of
leaving it with the code that raises the error.

Greetings,

Andres Freund

#2Jeff Davis
pgsql@j-davis.com
In reply to: Andres Freund (#1)
Re: PANIC serves too many masters

Hi,

On Sat, 2023-11-18 at 14:29 -0800, Andres Freund wrote:

I don't quite know what we should do. But the current situation
decidedly
doesn't seem great.

Agreed. Better classification is nice, but it also requires more
discipline and it might not always be obvious which category something
fits in. What about an error loop resulting in:

ereport(PANIC, (errmsg_internal("ERRORDATA_STACK_SIZE exceeded")));

We'd want a core file, but I don't think we want to restart in that
case, right?

Also, can we do a change like this incrementally by updating a few
PANIC sites at a time? Is it fine to leave plain PANICs in place for
the foreseeable future, or do you want all of them to eventually move?

Regards,
Jeff Davis

#3Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Davis (#2)
Re: PANIC serves too many masters

Jeff Davis <pgsql@j-davis.com> writes:

On Sat, 2023-11-18 at 14:29 -0800, Andres Freund wrote:

I don't quite know what we should do. But the current situation
decidedly
doesn't seem great.

Agreed.

+1

Better classification is nice, but it also requires more
discipline and it might not always be obvious which category something
fits in. What about an error loop resulting in:
ereport(PANIC, (errmsg_internal("ERRORDATA_STACK_SIZE exceeded")));
We'd want a core file, but I don't think we want to restart in that
case, right?

Why not restart? There's no strong reason to assume this will
repeat.

It might be worth having some independent logic in the postmaster
that causes it to give up after too many crashes in a row. But with
many/most of these call sites, by definition we're not too sure what
is wrong.

Also, can we do a change like this incrementally by updating a few
PANIC sites at a time? Is it fine to leave plain PANICs in place for
the foreseeable future, or do you want all of them to eventually move?

I'd be inclined to keep PANIC with its current meaning, and
incrementally change call sites where we decide that's not the
best behavior. I think those will be a minority, maybe a small
minority. (PANIC_EXIT had darn well better be a small minority.)

regards, tom lane

#4Jeff Davis
pgsql@j-davis.com
In reply to: Tom Lane (#3)
Re: PANIC serves too many masters

On Mon, 2023-11-20 at 17:12 -0500, Tom Lane wrote:

I'd be inclined to keep PANIC with its current meaning, and
incrementally change call sites where we decide that's not the
best behavior.  I think those will be a minority, maybe a small
minority.  (PANIC_EXIT had darn well better be a small minority.)

Is the error level the right way to express what we want to happen? It
seems like what we really want is to decide on the behavior, i.e.
restart or not, and generate core or not. That could be done a
different way, like:

ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

Regards,
Jeff Davis

#5Tom Lane
tgl@sss.pgh.pa.us
In reply to: Jeff Davis (#4)
Re: PANIC serves too many masters

Jeff Davis <pgsql@j-davis.com> writes:

Is the error level the right way to express what we want to happen? It
seems like what we really want is to decide on the behavior, i.e.
restart or not, and generate core or not. That could be done a
different way, like:

ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

Yeah, I was wondering about that too. It feels to me that
PANIC_EXIT is an error level (even more severe than PANIC).
But maybe "no core dump please" should be conveyed separately,
since it's just a minor adjustment that doesn't fundamentally
change what happens. It's plausible that you'd want a core,
or not want one, for different cases that all seem to require
PANIC_EXIT.

(Need a better name than PANIC_EXIT. OMIGOD?)

regards, tom lane

#6Andres Freund
andres@anarazel.de
In reply to: Tom Lane (#5)
Re: PANIC serves too many masters

Hi,

On 2023-11-20 17:55:32 -0500, Tom Lane wrote:

Jeff Davis <pgsql@j-davis.com> writes:

Is the error level the right way to express what we want to happen? It
seems like what we really want is to decide on the behavior, i.e.
restart or not, and generate core or not. That could be done a
different way, like:

ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

Yeah, I was wondering about that too. It feels to me that
PANIC_EXIT is an error level (even more severe than PANIC).
But maybe "no core dump please" should be conveyed separately,
since it's just a minor adjustment that doesn't fundamentally
change what happens.

I guess I was thinking of an error level because that'd be easier to search
for in logs. It seems reasonable to want to specificially search for errors
that cause core dumps, since IMO they should all be "should never happen" kind
of paths.

It's plausible that you'd want a core,
or not want one, for different cases that all seem to require
PANIC_EXIT.

I can't immediately think of a case where you'd want PANIC_EXIT but also want
a core dump? In my mental model to use PANIC_EXIT we'd need to have a decent
understanding that the situation isn't going to change after crash-restart -
in which case a core dump presumably isn't interesting?

(Need a better name than PANIC_EXIT. OMIGOD?)

CRITICAL?

I agree with the point made upthread that we'd want leave PANIC around, it's
not realistic to annotate everything, and then there's obviously also
extensions (although I hope there aren't many PANICs in extensions).

If that weren't the case, something like this could make sense:

PANIC: crash-restart
CRITICAL: crash-shutdown
BUG: crash-restart, abort()

Greetings,

Andres Freund

#7Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Jeff Davis (#4)
Re: PANIC serves too many masters

Hi,

(Coming back to this after looking at a bunch of core dumps from
disk-full PANICs.)

On Sat, 12 Sept 2026 at 03:10, Jeff Davis <pgsql@j-davis.com> wrote:

On Mon, 2023-11-20 at 17:12 -0500, Tom Lane wrote:

I'd be inclined to keep PANIC with its current meaning, and
incrementally change call sites where we decide that's not the
best behavior. I think those will be a minority, maybe a small
minority. (PANIC_EXIT had darn well better be a small minority.)

Is the error level the right way to express what we want to happen? It
seems like what we really want is to decide on the behavior, i.e.
restart or not, and generate core or not. That could be done a
different way, like:

ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

I gave the core-dump part of this a try. Two WIP patches attached.

I first thought of adding PANIC_NO_CORE, but wasn't sure where to put
it. Below PANIC, we'd have to adjust checks like elevel >= PANIC.
Above PANIC, it could take precedence over an ordinary PANIC during
nested error reporting. Neither seemed quite right when all I wanted
was to avoid the core dump.

Following your suggestion, I kept PANIC and added a flag to ErrorData.
The call sites use errnocoredump_on_errno(ENOSPC), which checks the
saved errno. For a marked PANIC, errfinish() takes the _exit(2) path
instead of calling abort(). There's also a developer GUC to get the
abort() behavior back when needed. (may not be needed?)

The second patch adds this to selected WAL and pg_control error paths.
I haven't changed the restart policy in any of them, patch is just for the
core dump behaviour.

Would something along these lines make sense?

Regards,
Ayush

Attachments:

t48753_7
v1-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchapplication/octet-stream; name=v1-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchDownload+469-3
v1-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchapplication/octet-stream; name=v1-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchDownload+27-1
#8Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Ayush Tiwari (#7)
Re: PANIC serves too many masters

Hi,

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com>
wrote:

Hi,

(Coming back to this after looking at a bunch of core dumps from
disk-full PANICs.)

On Sat, 12 Sept 2026 at 03:10, Jeff Davis <pgsql@j-davis.com> wrote:

On Mon, 2023-11-20 at 17:12 -0500, Tom Lane wrote:

I'd be inclined to keep PANIC with its current meaning, and
incrementally change call sites where we decide that's not the
best behavior. I think those will be a minority, maybe a small
minority. (PANIC_EXIT had darn well better be a small minority.)

Is the error level the right way to express what we want to happen? It
seems like what we really want is to decide on the behavior, i.e.
restart or not, and generate core or not. That could be done a
different way, like:

ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

I gave the core-dump part of this a try. Two WIP patches attached.

I first thought of adding PANIC_NO_CORE, but wasn't sure where to put
it. Below PANIC, we'd have to adjust checks like elevel >= PANIC.
Above PANIC, it could take precedence over an ordinary PANIC during
nested error reporting. Neither seemed quite right when all I wanted
was to avoid the core dump.

Following your suggestion, I kept PANIC and added a flag to ErrorData.
The call sites use errnocoredump_on_errno(ENOSPC), which checks the
saved errno. For a marked PANIC, errfinish() takes the _exit(2) path
instead of calling abort(). There's also a developer GUC to get the
abort() behavior back when needed. (may not be needed?)

The second patch adds this to selected WAL and pg_control error paths.
I haven't changed the restart policy in any of them, patch is just for the
core dump behaviour.

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

The repeated crash/recovery cycles hit the unrelated io_uring resource
leak discussed at [1]/messages/by-id/zHN8Rb7syWYDBPUhCICxKLr_tQmFLg0T024irxVVb6RSSicZcqcA0eT73z2PQy7JmzzqnUzX0Hi3P2dPKRvmNx_jbAqSCLyqJvWjXEypyAI=@draescher.fr in CFBot. Using worker avoids that failure; the
rest of the code is unchanged from v1.

[1]: /messages/by-id/zHN8Rb7syWYDBPUhCICxKLr_tQmFLg0T024irxVVb6RSSicZcqcA0eT73z2PQy7JmzzqnUzX0Hi3P2dPKRvmNx_jbAqSCLyqJvWjXEypyAI=@draescher.fr
/messages/by-id/zHN8Rb7syWYDBPUhCICxKLr_tQmFLg0T024irxVVb6RSSicZcqcA0eT73z2PQy7JmzzqnUzX0Hi3P2dPKRvmNx_jbAqSCLyqJvWjXEypyAI=@draescher.fr

Regards,
Ayush

Attachments:

t48753_8
v2-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchapplication/octet-stream; name=v2-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchDownload+27-1
v2-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchapplication/octet-stream; name=v2-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchDownload+471-3
#9Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Ayush Tiwari (#8)
Re: PANIC serves too many masters

Hi,

On Fri, 18 Sept 2026 at 20:13, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

I gave the core-dump part of this a try. Two WIP patches attached.

I first thought of adding PANIC_NO_CORE, but wasn't sure where to put
it. Below PANIC, we'd have to adjust checks like elevel >= PANIC.
Above PANIC, it could take precedence over an ordinary PANIC during
nested error reporting. Neither seemed quite right when all I wanted
was to avoid the core dump.

I found only one place which does 'elevel >= PANIC', in elog.c:

if (elevel >= PANIC)
{
/*
* Serious crash time. Postmaster will observe SIGABRT process exit
* status and kill the other backends too.
*
* XXX: what if we are *in* the postmaster? abort() won't kill our
* children...
*/
fflush(NULL);
abort();
}

I think this place is easy to change, but determining when to core
dump (i.e. setting the correct error level) might be hard without
looking at the error code itself. We can set the error level by
looking at the error code, that might be another approach.

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

I have reviewed v2-0001 of this patch and have a couple of comments.

1. What do you think about generalizing errnocoredump_on_errno()
function with errabort($bool) like what Jeff suggested [1]ereport(PANIC, (errmsg("could not locate a valid checkpoint record"), errabort(false),errrestart(false))); upthread?
Using a more general function will work better than passing errno to a
specific function, IMO.

2. The test coverage looks useful but I think it is too complicated
for this feature. Could we simplify the tests? Also, I am not sure we
need tests for this at all, but I don't have a strong opinion.

3.

@@ -611,13 +624,16 @@ errfinish(const char *filename, int lineno,
const char *funcname)
  if (elevel >= PANIC)
  {
  /*
- * Serious crash time. Postmaster will observe SIGABRT process exit
- * status and kill the other backends too.
+ * Serious crash time.  If this error is marked not to dump core, exit
+ * without running cleanup callbacks.  Exit code 2 makes the postmaster
+ * treat this as a crash and kill the other backends too.
  *
  * XXX: what if we are *in* the postmaster?  abort() won't kill our
  * children...
  */

Could this comment mention both termination paths? Exit code 2 and
termination by SIGABRT both trigger the postmaster's crash handling.
The revised wording explains the former but omits the explanation of
the latter.

4.

@@ -1652,6 +1668,22 @@ errhidecontext(bool hide_ctx)
return 0; /* return value does not matter */
}

+/*
+ * errnocoredump_on_errno --- skip a core dump for the specified errno
+ */
+int
+errnocoredump_on_errno(int errnum)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ edata->no_core_dump |= edata->saved_errno == errnum;

Once no_core_dump is set to true, we can't go back to false; which
means we can't force core dumps for specific errors. I am not sure
this is a problem now, but I think this is a limitation worth
considering.

[1]: ereport(PANIC, (errmsg("could not locate a valid checkpoint record"), errabort(false),errrestart(false)));
ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

--
Regards,
Nazir Bilal Yavuz
Microsoft

#10Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Nazir Bilal Yavuz (#9)
Re: PANIC serves too many masters

Hi,

Thanks for the review!

On Fri, 2 Oct 2026 at 13:36, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Fri, 18 Sept 2026 at 20:13, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

I gave the core-dump part of this a try. Two WIP patches attached.

I first thought of adding PANIC_NO_CORE, but wasn't sure where to put
it. Below PANIC, we'd have to adjust checks like elevel >= PANIC.
Above PANIC, it could take precedence over an ordinary PANIC during
nested error reporting. Neither seemed quite right when all I wanted
was to avoid the core dump.

I found only one place which does 'elevel >= PANIC', in elog.c:

if (elevel >= PANIC)
{
/*
* Serious crash time. Postmaster will observe SIGABRT process exit
* status and kill the other backends too.
*
* XXX: what if we are *in* the postmaster? abort() won't kill our
* children...
*/
fflush(NULL);
abort();
}

I think this place is easy to change, but determining when to core
dump (i.e. setting the correct error level) might be hard without
looking at the error code itself. We can set the error level by
looking at the error code, that might be another approach.

I had errstart() in mind too, where it promotes errors in critical
sections and takes the Max() over the stack. So v3 keeps the levels as
they are and only controls whether we call abort().

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

I have reviewed v2-0001 of this patch and have a couple of comments.

1. What do you think about generalizing errnocoredump_on_errno()
function with errabort($bool) like what Jeff suggested [1] upthread?
Using a more general function will work better than passing errno to a
specific function, IMO.

Added errabort(bool), but kept the errno helper for the WAL callers.
AFAICS dropping it would mean saving errno at each call site, since
ereport() doesn't allow it in the argument list. Do you think the
wrapper is worth keeping, or should the callers do that themselves?

2. The test coverage looks useful but I think it is too complicated
for this feature. Could we simplify the tests? Also, I am not sure we
need tests for this at all, but I don't have a strong opinion.

I pulled out the repeated crash/restart checks and removed the unused
branches in the callbacks.
I think we should keep the promoted ERROR, copied ErrorData and nested
PANIC cases?

3.

@@ -611,13 +624,16 @@ errfinish(const char *filename, int lineno,
const char *funcname)
if (elevel >= PANIC)
{
/*
- * Serious crash time. Postmaster will observe SIGABRT process exit
- * status and kill the other backends too.
+ * Serious crash time.  If this error is marked not to dump core, exit
+ * without running cleanup callbacks.  Exit code 2 makes the postmaster
+ * treat this as a crash and kill the other backends too.
*
* XXX: what if we are *in* the postmaster?  abort() won't kill our
* children...
*/

Could this comment mention both termination paths? Exit code 2 and
termination by SIGABRT both trigger the postmaster's crash handling.
The revised wording explains the former but omits the explanation of
the latter.

Thanks, done.

4.

@@ -1652,6 +1668,22 @@ errhidecontext(bool hide_ctx)
return 0; /* return value does not matter */
}

+/*
+ * errnocoredump_on_errno --- skip a core dump for the specified errno
+ */
+int
+errnocoredump_on_errno(int errnum)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ edata->no_core_dump |= edata->saved_errno == errnum;

Once no_core_dump is set to true, we can't go back to false; which
means we can't force core dumps for specific errors. I am not sure
this is a problem now, but I think this is a limitation worth
considering.

The |= in errnocoredump_on_errno() was so that a later non-matching
call wouldn't undo an earlier match. In v3 errabort() sets
no_core_dump directly, so errabort(true) can undo an earlier
errabort(false), and there's a test for that. The check in errfinish()
is unchanged though, since a nested error shouldn't suppress the core
we'd have taken for the original PANIC. Thoughts?

PFA v3, with latest updates.

Regards,
Ayush

Show quoted text

[1]
ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

Attachments:

t48753_10
v3-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchapplication/octet-stream; name=v3-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchDownload+413-4
v3-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchapplication/octet-stream; name=v3-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchDownload+27-1
#11Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Ayush Tiwari (#10)
Re: PANIC serves too many masters

Hi,

On Fri, 2 Oct 2026 at 16:38, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Fri, 2 Oct 2026 at 13:36, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Fri, 18 Sept 2026 at 20:13, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

I gave the core-dump part of this a try. Two WIP patches attached.

I first thought of adding PANIC_NO_CORE, but wasn't sure where to put
it. Below PANIC, we'd have to adjust checks like elevel >= PANIC.
Above PANIC, it could take precedence over an ordinary PANIC during
nested error reporting. Neither seemed quite right when all I wanted
was to avoid the core dump.

I found only one place which does 'elevel >= PANIC', in elog.c:

if (elevel >= PANIC)
{
/*
* Serious crash time. Postmaster will observe SIGABRT process exit
* status and kill the other backends too.
*
* XXX: what if we are *in* the postmaster? abort() won't kill our
* children...
*/
fflush(NULL);
abort();
}

I think this place is easy to change, but determining when to core
dump (i.e. setting the correct error level) might be hard without
looking at the error code itself. We can set the error level by
looking at the error code, that might be another approach.

I had errstart() in mind too, where it promotes errors in critical
sections and takes the Max() over the stack. So v3 keeps the levels as
they are and only controls whether we call abort().

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

I have reviewed v2-0001 of this patch and have a couple of comments.

1. What do you think about generalizing errnocoredump_on_errno()
function with errabort($bool) like what Jeff suggested [1] upthread?
Using a more general function will work better than passing errno to a
specific function, IMO.

Added errabort(bool), but kept the errno helper for the WAL callers.
AFAICS dropping it would mean saving errno at each call site, since
ereport() doesn't allow it in the argument list. Do you think the
wrapper is worth keeping, or should the callers do that themselves?

2. The test coverage looks useful but I think it is too complicated
for this feature. Could we simplify the tests? Also, I am not sure we
need tests for this at all, but I don't have a strong opinion.

I pulled out the repeated crash/restart checks and removed the unused
branches in the callbacks.
I think we should keep the promoted ERROR, copied ErrorData and nested
PANIC cases?

3.

@@ -611,13 +624,16 @@ errfinish(const char *filename, int lineno,
const char *funcname)
if (elevel >= PANIC)
{
/*
- * Serious crash time. Postmaster will observe SIGABRT process exit
- * status and kill the other backends too.
+ * Serious crash time.  If this error is marked not to dump core, exit
+ * without running cleanup callbacks.  Exit code 2 makes the postmaster
+ * treat this as a crash and kill the other backends too.
*
* XXX: what if we are *in* the postmaster?  abort() won't kill our
* children...
*/

Could this comment mention both termination paths? Exit code 2 and
termination by SIGABRT both trigger the postmaster's crash handling.
The revised wording explains the former but omits the explanation of
the latter.

Thanks, done.

4.

@@ -1652,6 +1668,22 @@ errhidecontext(bool hide_ctx)
return 0; /* return value does not matter */
}

+/*
+ * errnocoredump_on_errno --- skip a core dump for the specified errno
+ */
+int
+errnocoredump_on_errno(int errnum)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ edata->no_core_dump |= edata->saved_errno == errnum;

Once no_core_dump is set to true, we can't go back to false; which
means we can't force core dumps for specific errors. I am not sure
this is a problem now, but I think this is a limitation worth
considering.

The |= in errnocoredump_on_errno() was so that a later non-matching
call wouldn't undo an earlier match. In v3 errabort() sets
no_core_dump directly, so errabort(true) can undo an earlier
errabort(false), and there's a test for that. The check in errfinish()
is unchanged though, since a nested error shouldn't suppress the core
we'd have taken for the original PANIC. Thoughts?

PFA v3, with latest updates.

I've trimmed the test code quite a bit. There is just one helper now,
with three cases: ENOSPC skips abort(), EIO still calls it, and the
override forces abort() for ENOSPC.

For ease of review, I've put the tests in 0003, leaving 0001 and 0002
focused on the core and WAL changes. The core and WAL code is unchanged
from v3.

v4 attached.

Regards,
Ayush

Show quoted text

[1]
ereport(PANIC,
(errmsg("could not locate a valid checkpoint record"),
errabort(false),errrestart(false)));

Attachments:

t48753_11
v4-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchapplication/octet-stream; name=v4-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchDownload+110-4
v4-0003-Test-PANIC-core-dump-suppression.patchapplication/octet-stream; name=v4-0003-Test-PANIC-core-dump-suppression.patchDownload+105-1
v4-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchapplication/octet-stream; name=v4-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchDownload+27-1
#12Nazir Bilal Yavuz
byavuz81@gmail.com
In reply to: Ayush Tiwari (#11)
Re: PANIC serves too many masters

Hi,

On Mon, 12 Oct 2026 at 10:04, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Fri, 2 Oct 2026 at 16:38, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Fri, 2 Oct 2026 at 13:36, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Fri, 18 Sept 2026 at 20:13, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

I have reviewed v2-0001 of this patch and have a couple of comments.

1. What do you think about generalizing errnocoredump_on_errno()
function with errabort($bool) like what Jeff suggested [1] upthread?
Using a more general function will work better than passing errno to a
specific function, IMO.

Added errabort(bool), but kept the errno helper for the WAL callers.
AFAICS dropping it would mean saving errno at each call site, since
ereport() doesn't allow it in the argument list. Do you think the
wrapper is worth keeping, or should the callers do that themselves?

I think keeping it as a wrapper makes sense. It uses saved_errno and
avoids having each WAL caller save errno before ereport(). That seems
nice to me.

2. The test coverage looks useful but I think it is too complicated
for this feature. Could we simplify the tests? Also, I am not sure we
need tests for this at all, but I don't have a strong opinion.

I pulled out the repeated crash/restart checks and removed the unused
branches in the callbacks.
I think we should keep the promoted ERROR, copied ErrorData and nested
PANIC cases?

That looks good. One small coverage point: what aboud adding a case
which suppresses the core dump and then calls errabort(true), with the
force_core_dump_on_panic GUC off?

4.

@@ -1652,6 +1668,22 @@ errhidecontext(bool hide_ctx)
return 0; /* return value does not matter */
}

+/*
+ * errnocoredump_on_errno --- skip a core dump for the specified errno
+ */
+int
+errnocoredump_on_errno(int errnum)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ edata->no_core_dump |= edata->saved_errno == errnum;

Once no_core_dump is set to true, we can't go back to false; which
means we can't force core dumps for specific errors. I am not sure
this is a problem now, but I think this is a limitation worth
considering.

The |= in errnocoredump_on_errno() was so that a later non-matching
call wouldn't undo an earlier match. In v3 errabort() sets
no_core_dump directly, so errabort(true) can undo an earlier
errabort(false), and there's a test for that. The check in errfinish()
is unchanged though, since a nested error shouldn't suppress the core
we'd have taken for the original PANIC. Thoughts?

LGTM.

For ease of review, I've put the tests in 0003, leaving 0001 and 0002
focused on the core and WAL changes. The core and WAL code is unchanged
from v3.

Thanks, this version is easier to review.

Code comments:

0001:

1.

+/*
+ * errabort --- set whether a PANIC should call abort()
+ */
+int
+errabort(bool do_abort)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ CHECK_STACK_DEPTH();
+ edata->no_core_dump = !do_abort;
+
+ return 0;
+}

It is cosmetic but it seems you forgot these comments:

/* we don't bother incrementing recursion_depth */
and
/* return value does not matter */

0002: Changes look correct but perhaps you can add
errnocoredump_on_errno(ENOSPC) to XLogWalRcvWrite() in walreceiver.c.

0003:

1.

+static void
+#ifdef WIN32
+injection_abort_handler(int signal)
+#else
+injection_abort_handler(SIGNAL_ARGS)
+#endif
+{
+ _exit(42);
+}
+

I think this function needs a comment. It is hard to understand this
function's purpose and also the special number.

2.

+/* Make abort() observable without producing a core file. */
+PG_FUNCTION_INFO_V1(injection_points_panic);

This comment doesn't seem relevant to the EIO input. Something like
this 'Test PANIC core-dump suppression with ENOSPC or EIO.' can fit
better. Also, function name is a bit general, what about something
like 'injection_points_test_panic_core_dump'?

--
Regards,
Nazir Bilal Yavuz
Microsoft

#13Ayush Tiwari
ayushtiwari.slg01@gmail.com
In reply to: Nazir Bilal Yavuz (#12)
Re: PANIC serves too many masters

Hi,

On Mon, 12 Oct 2026 at 17:34, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

Hi,

On Mon, 12 Oct 2026 at 10:04, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Fri, 2 Oct 2026 at 16:38, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Fri, 2 Oct 2026 at 13:36, Nazir Bilal Yavuz <byavuz81@gmail.com> wrote:

On Fri, 18 Sept 2026 at 20:13, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

On Sat, 12 Sept 2026 at 21:50, Ayush Tiwari <ayushtiwari.slg01@gmail.com> wrote:

Would something along these lines make sense?

Attached is v2, which sets io_method=worker for the TAP test.

I have reviewed v2-0001 of this patch and have a couple of comments.

1. What do you think about generalizing errnocoredump_on_errno()
function with errabort($bool) like what Jeff suggested [1] upthread?
Using a more general function will work better than passing errno to a
specific function, IMO.

Added errabort(bool), but kept the errno helper for the WAL callers.
AFAICS dropping it would mean saving errno at each call site, since
ereport() doesn't allow it in the argument list. Do you think the
wrapper is worth keeping, or should the callers do that themselves?

I think keeping it as a wrapper makes sense. It uses saved_errno and
avoids having each WAL caller save errno before ereport(). That seems
nice to me.

Thanks for the review!

2. The test coverage looks useful but I think it is too complicated
for this feature. Could we simplify the tests? Also, I am not sure we
need tests for this at all, but I don't have a strong opinion.

I pulled out the repeated crash/restart checks and removed the unused
branches in the callbacks.
I think we should keep the promoted ERROR, copied ErrorData and nested
PANIC cases?

That looks good. One small coverage point: what aboud adding a case
which suppresses the core dump and then calls errabort(true), with the
force_core_dump_on_panic GUC off?

Ahh yes, added, thanks.

4.

@@ -1652,6 +1668,22 @@ errhidecontext(bool hide_ctx)
return 0; /* return value does not matter */
}

+/*
+ * errnocoredump_on_errno --- skip a core dump for the specified errno
+ */
+int
+errnocoredump_on_errno(int errnum)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ /* we don't bother incrementing recursion_depth */
+ CHECK_STACK_DEPTH();
+
+ edata->no_core_dump |= edata->saved_errno == errnum;

Once no_core_dump is set to true, we can't go back to false; which
means we can't force core dumps for specific errors. I am not sure
this is a problem now, but I think this is a limitation worth
considering.

The |= in errnocoredump_on_errno() was so that a later non-matching
call wouldn't undo an earlier match. In v3 errabort() sets
no_core_dump directly, so errabort(true) can undo an earlier
errabort(false), and there's a test for that. The check in errfinish()
is unchanged though, since a nested error shouldn't suppress the core
we'd have taken for the original PANIC. Thoughts?

LGTM.

Code comments:

0001:

1.

+/*
+ * errabort --- set whether a PANIC should call abort()
+ */
+int
+errabort(bool do_abort)
+{
+ ErrorData  *edata = &errordata[errordata_stack_depth];
+
+ CHECK_STACK_DEPTH();
+ edata->no_core_dump = !do_abort;
+
+ return 0;
+}

It is cosmetic but it seems you forgot these comments:

/* we don't bother incrementing recursion_depth */
and
/* return value does not matter */

My bad, added them in v5.

0002: Changes look correct but perhaps you can add
errnocoredump_on_errno(ENOSPC) to XLogWalRcvWrite() in walreceiver.c.

Yes, that write-error PANIC should get the same treatment. Added it.

0003:

1.

+static void
+#ifdef WIN32
+injection_abort_handler(int signal)
+#else
+injection_abort_handler(SIGNAL_ARGS)
+#endif
+{
+ _exit(42);
+}
+

I think this function needs a comment. It is hard to understand this
function's purpose and also the special number.

Done.

2.

+/* Make abort() observable without producing a core file. */
+PG_FUNCTION_INFO_V1(injection_points_panic);

This comment doesn't seem relevant to the EIO input. Something like
this 'Test PANIC core-dump suppression with ENOSPC or EIO.' can fit
better. Also, function name is a bit general, what about something
like 'injection_points_test_panic_core_dump'?

Hmm, that sounds better, I've renamed it as you suggested.

v5 attached with all the above changes.

Regards,
Ayush

Attachments:

t48753_13
v5-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchapplication/octet-stream; name=v5-0001-Allow-selected-PANIC-errors-to-skip-core-dumps.patchDownload+112-4
v5-0003-Test-PANIC-core-dump-suppression.patchapplication/octet-stream; name=v5-0003-Test-PANIC-core-dump-suppression.patchDownload+114-1
v5-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchapplication/octet-stream; name=v5-0002-Avoid-core-dumps-for-WAL-disk-full-failures.patchDownload+28-1