Re: Return pg_control from pg_backup_stop().

Started by Greg Burd24 days ago6 messageshackers
Jump to latest
#1Greg Burd
gregburd@amazon.com

On Thu, 29 Oct 2026 09:50:46 +0900, Michael Paquier <michael@paquier.xyz> wrote:

On Wed, Oct 28, 2026 at 03:38:55PM +0900, Michael Paquier wrote:

My blind suggestion of fix would be the attached, that I cannot really
test. Thoughts or comments?

I unfortunately do not have a better idea after sleeping on it, so I
have applied this palloc0/pfree solution. Now I'll go light a candle
to make the buildfarm gods kinder to me.
--
Michael

Hey Michael,

As requested (back channel) I spent some quality time with my
UltraSPARC 45 this morning and dug into this.

TL;DR: it's a GCC bug, and c97931c16d fixes it for us by keeping
perform_base_backup()'s frame under 4kB.

I reproduced it on an UltraSPARC-IIIi running illumos with gcc 13.4,
-O2 and cassert, at 4a92c8c29b. Your suggested reproduction works: a
file with a 114-byte name in PGDATA, then BASE_BACKUP through psql
with "replication=1", gives the SIGSEGV every time. So does
cancelling a BASE_BACKUP (MAX_RATE 32), which is thorntail's case.
The core (mdb, full output attached):

status: process terminated by SIGSEGV, addr=fffffffe7f74733c
%pc = perform_base_backup+0x1c0: st %o0, [%g1]
%npc = perform_base_backup+0x1480
%i0 = 0x00000001008b6000
perform_base_backup+0x1c0(...)
SendBaseBackup+0x5a0(...)
exec_replication_command+0x5b4(...)
PostgresMain+0x8f0(...)

That store is in the delay slot of the branch right after
sigsetjmp() in PG_TRY(), on its second return. Here is the
disassembly you asked for, at that spot:

227de22095 (before), frame 2224 bytes:
call sigsetjmp
...
bne,pn %icc, <catch>
st %o0, [ %fp + 0x57 ]

4a92c8c29b, frame 10368 bytes:
sethi %hi(0x1c00), %i0 ! in the prologue
...
call sigsetjmp
xor %i0, -837, %g1
add %g1, %fp, %g1
bne,pn %icc, <catch>
st %o0, [ %g1 ]

c97931c16d (fix), frame 2096 bytes: same as before.

With the 8kB array, the frame slots are more than 4kB from %fp, too
far for SPARC's 13-bit immediate offset, so GCC needs a register to
hold the offset. It keeps the constant 0x1c00 in %i0 across the
sigsetjmp() call and reuses it afterwards, and later in the PG_TRY
block it loads a string address into %i0. On SPARC, siglongjmp()
doesn't restore %l and %i registers from the jmp_buf; it resumes the
register window from the frame's save area, so %i0 comes back with
the value it had at siglongjmp() time. In the core, %i0 is the
string address instead of 0x1c00, and (0x1008b6000 ^ -837) + %fp is
exactly the faulting address. On other platforms jmp_buf saves the
callee-saved registers, so the constant would come back intact. The
garbage address explains SIGSEGV on billbug and the occasional
SIGBUS on thorntail, and why the WARNING from do_pg_abort_backup()
is never logged: the backend dies before the catch block runs.

The GCC RTL dumps show where it goes wrong. LRA does reload the
constant after the call, (set (reg %i0) (const_int 7168)), but
postreload's reload_cse_move2add() turns that into a no-op move,
because it thinks %i0 survives the call. cselib resets its state at
a REG_SETJMP call, but move2add has no such check, on GCC master as
well.

To rule out Postgres entirely, the attached sj.c/sj2.c is a small
program with no Postgres code: a function with an 8kB local array and
a sigsetjmp(). With gcc 13.4 -O2 it segfaults on this machine, and
it does the same when built with Debian's sparc64-linux-gnu-gcc-13
and run under qemu. It runs fine at -O1 and -O0, and with gcc 4.4.
I'm not sure if this has been, or should be, reported to GCC as yet.

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs. With it reverted, the same TAP run
logs the SIGSEGV. The test still reports PASS in that case because
it only checks that pg_basebackup fails.

I don't see a better fix on our side. A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB. In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]). Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation. I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

best.

-greg

Attachments:

sj.capplication/octet-stream; name=sj.cDownload
sj2.capplication/octet-stream; name=sj2.cDownload
#2Michael Paquier
michael@paquier.xyz
In reply to: Greg Burd (#1)

On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs. With it reverted, the same TAP run
logs the SIGSEGV. The test still reports PASS in that case because
it only checks that pg_basebackup fails.

Ah, thanks for the in-depth investigation. So the previous guess was
really right. Candles work.

I don't see a better fix on our side. A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB. In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]). Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation. I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

Ugh. Even if the buffer itself is not used within the try/catch
block itself? Getting entirely rid of the TRY/CATCH would lead to a
much nicer result overall: we'd be able to keep the local buffer. Or
perhaps go to the simpler solution of a palloc/pfree? I cannot
believe that this extra allocation would matter for lo_export() calls.

If you are proved to be right, it sounds to me that we ought to
fix and backpatch that as well, especially if there is nothing we can
do on the Postgres side. I don't believe in luck and magic when it
comes to software programming.

Thoughts?
--
Michael

#3David Steele
david@pgbackrest.org
In reply to: Michael Paquier (#2)

On 10/30/26 00:08, Michael Paquier wrote:

On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs. With it reverted, the same TAP run
logs the SIGSEGV. The test still reports PASS in that case because
it only checks that pg_basebackup fails.

Ah, thanks for the in-depth investigation. So the previous guess was
really right. Candles work.

Thank you, Greg! This is exactly the confirmation we needed.

I would encourage you to file a bug with gcc. I've found a few bugs and
the gcc folks are really easy to work with. Since you have a repro it
will be easy to produce the pre-processed output they want.

One bad thing -- even if they fix the bug, distros may choose not to
back patch it (so it may already be fixed). I had to maintain a patch
for a gcc bug on ppc64le for years before the affected distros EOL'd.
That bug limited where I could use const, which was pretty annoying.

I don't see a better fix on our side. A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB. In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]). Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation. I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

Ugh. Even if the buffer itself is not used within the try/catch
block itself?

Apparently so -- the failure of the base backup cancel test showed that.
The buffer was never used and just the larger frame caused the problem.

Getting entirely rid of the TRY/CATCH would lead to a
much nicer result overall: we'd be able to keep the local buffer. Or
perhaps go to the simpler solution of a palloc/pfree? I cannot
believe that this extra allocation would matter for lo_export() calls.

palloc/pfree seems like the simpler solution.

If you are proved to be right, it sounds to me that we ought to
fix and backpatch that as well, especially if there is nothing we can
do on the Postgres side. I don't believe in luck and magic when it
comes to software programming.

I've attached a patch for be_lo_export(). Working around compiler bugs
is never fun but I'm not sure what choice we have.

Regards,
-David

Attachments:

t254447_3
lo-export-heap-buffer.patchtext/plain; charset=UTF-8; name=lo-export-heap-buffer.patchDownload+10-1
#4Greg Burd
greg@burd.me
In reply to: Michael Paquier (#2)

On Thursday, October 29th, 2026 at 7:08 PM, Michael Paquier <michael@paquier.xyz> wrote:

On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs. With it reverted, the same TAP run
logs the SIGSEGV. The test still reports PASS in that case because
it only checks that pg_basebackup fails.

Ah, thanks for the in-depth investigation.

Hey Michael, sure thing. It's why I bought the UltraSPARC 45, oddly
enough!

So the previous guess was really right.

Yes, you have a talent for many things including guessing fixes to
obscure issues.

Candles work.

Indeed they do. :)

I don't see a better fix on our side. A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB. In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]). Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation. I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

Ugh. Even if the buffer itself is not used within the try/catch
block itself?

I could test if you'd like.

Getting entirely rid of the TRY/CATCH would lead to a
much nicer result overall: we'd be able to keep the local buffer. Or
perhaps go to the simpler solution of a palloc/pfree? I cannot
believe that this extra allocation would matter for lo_export() calls.

If you are proved to be right, it sounds to me that we ought to
fix and backpatch that as well, especially if there is nothing we can
do on the Postgres side.

Better safe than sorry. Even if I submit a bug to GCC and they fix it
there it will take time to get that shipped and you have people who rarely
update, etc. etc.

I don't believe in luck and magic when it comes to software programming.

Nope, just 1s and 0s.

Thoughts?
--
Michael

best.

-greg

#5Greg Burd
greg@burd.me
In reply to: David Steele (#3)

On Friday, October 30th, 2026 at 4:34 AM, David Steele <david@pgbackrest.org> wrote:

On 10/30/26 00:08, Michael Paquier wrote:

On Thu, Oct 29, 2026 at 06:11:39PM +0000, Burd, Greg wrote:

Validation of c97931c16d on the same machine: with it applied, three
runs each of both reproductions give the WARNING and ERROR and no
crash, and t/010_pg_basebackup.pl passes all 168 tests with no
backend crash in the server logs. With it reverted, the same TAP run
logs the SIGSEGV. The test still reports PASS in that case because
it only checks that pg_basebackup fails.

Ah, thanks for the in-depth investigation. So the previous guess was
really right. Candles work.

Thank you, Greg! This is exactly the confirmation we needed.

Hey David, happy to help. As I said to Michael, now I can write off that
UltraSPARC 45 and the power it gobbles up on my taxes with proof that it
is used for work.

I would encourage you to file a bug with gcc. I've found a few bugs and
the gcc folks are really easy to work with. Since you have a repro it
will be easy to produce the pre-processed output they want.

Sure, I'll carve out some time to do that.

One bad thing -- even if they fix the bug, distros may choose not to
back patch it (so it may already be fixed). I had to maintain a patch
for a gcc bug on ppc64le for years before the affected distros EOL'd.
That bug limited where I could use const, which was pretty annoying.

But that's the life of an open source developer, right? I signed up for
it so I can't complain. Or if I do, you should ignore me.

I don't see a better fix on our side. A helper function with
the array would also work, but palloc0() is simpler, and both rely on
the same thing: no PG_TRY() frame larger than 4kB. In a core build
on this machine, the only other function with a frame that large that
calls sigsetjmp() is be_lo_export() (char buf[BUFSIZE]). Its code
after sigsetjmp() reloads the constant, so it is fine today, but only
by luck of register allocation. I haven't checked contrib or other
GCC versions or as mentioned earlier against their bug system.

Ugh. Even if the buffer itself is not used within the try/catch
block itself?

Apparently so -- the failure of the base backup cancel test showed that.
The buffer was never used and just the larger frame caused the problem.

I agree.

Getting entirely rid of the TRY/CATCH would lead to a
much nicer result overall: we'd be able to keep the local buffer. Or
perhaps go to the simpler solution of a palloc/pfree? I cannot
believe that this extra allocation would matter for lo_export() calls.

palloc/pfree seems like the simpler solution.

+1

If you are proved to be right, it sounds to me that we ought to
fix and backpatch that as well, especially if there is nothing we can
do on the Postgres side. I don't believe in luck and magic when it
comes to software programming.

I've attached a patch for be_lo_export(). Working around compiler bugs
is never fun but I'm not sure what choice we have.

I've not reviewed the patch yet, but it is on my queue.

Regards,
-David

best.

-greg

#6Michael Paquier
michael@paquier.xyz
In reply to: Greg Burd (#5)

On Fri, Oct 30, 2026 at 04:04:13PM +0000, Greg Burd wrote:

On Friday, October 30th, 2026 at 4:34 AM, David Steele <david@pgbackrest.org> wrote:

I've attached a patch for be_lo_export(). Working around compiler bugs
is never fun but I'm not sure what choice we have.

I've not reviewed the patch yet, but it is on my queue.

Thanks. Making sure that be_lo_export() can be faulty on SPARC and
also making sure that the palloc/pfree fixes it would be really
useful. I'm not sure that I'm going to spend cycles spawning an
instance just for this purpose. That does not feel like the most
useful use of time with everything that is spawning on the lists these
days.
--
Michael