BUG #16951: pg_restore segfaults on custom format piped from a different version of PG
The following bug has been logged on the website:
Bug reference: 16951
Logged by: Sergey Koposov
Email address: skoposov@ed.ac.uk
PostgreSQL version: 10.16
Operating system: Linux
Description:
Hi,
I have a reproducible case of segfaulting pg_restore when trying to restore
from pg_dump of a different version. Specifically at least pg_restore from
10 crashes from pg_dump 12.
I understand that this is not supported, but presumably it still shouldn't
segfault.
This was a command
pg_dump12 -n SCHEMA -Fc -U dbadmin DB | pg_restore10 -U dbadmin -h
localhost -1 -d DB
where pg_dump12 is pg_dump from 12.6 on one linux 64bit machine and
pg_restore10 is pg_restore from 10.16 on another linux 64bit machine
I attach the gdb bt full of the crash (see below). I also have a 512 byte
file that crashes pg_restore (the top 512 bytes from the pgdump). I can
share it if needed.
It is clear that some checks of the version of the archive have not been
done early enough by pg_restore leading to the segfault. I don't have time
to get to the bottom of this, but
I'm seeing that readHead() in bg_backup_archiver() has not executed the
checks
behind
if (!AH->readHeader) that would have failed.
And it also looks like the readHeader flag is set early by
_discoverArchiveFormat()
on when reading from stdin.
(but this is just my impression from a quick look at the code)
Cheers,
Sergey
#0 __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
#1 0x00000000004098d0 in ReadToc (AH=0xe8bb60) at
pg_backup_archiver.c:2660
#2 0x000000000040f010 in InitArchiveFmt_Custom (AH=0xe8bb60)
at pg_backup_custom.c:191
#3 0x0000000000408f57 in _allocAH (FileSpec=0x0, fmt=archUnknown,
compression=0, dosync=1 '\001', mode=archModeRead,
setupWorkerPtr=0x404528 <setupRestoreWorker>) at
pg_backup_archiver.c:2400
#4 0x00000000004045d3 in OpenArchive (FileSpec=0x0, fmt=archUnknown)
at pg_backup_archiver.c:235
#5 0x0000000000403eff in main (argc=7, argv=0x7fffb0559018)
at pg_restore.c:400
#0 __strcmp_sse2_unaligned ()
at ../sysdeps/x86_64/multiarch/strcmp-sse2-unaligned.S:31
No locals.
#1 0x00000000004098d0 in ReadToc (AH=0xe8bb60) at
pg_backup_archiver.c:2660
i = 0
tmp = 0xe6d7d0 "ENCODING"
deps = 0x7fa9b5de08e0 <_IO_2_1_stdin_>
depIdx = 15127296
depSize = 0
te = 0xe91140
#2 0x000000000040f010 in InitArchiveFmt_Custom (AH=0xe8bb60)
at pg_backup_custom.c:191
ctx = 0xe8d100
#3 0x0000000000408f57 in _allocAH (FileSpec=0x0, fmt=archUnknown,
compression=0, dosync=1 '\001', mode=archModeRead,
setupWorkerPtr=0x404528 <setupRestoreWorker>) at
pg_backup_archiver.c:2400
AH = 0xe8bb60
#4 0x00000000004045d3 in OpenArchive (FileSpec=0x0, fmt=archUnknown)
at pg_backup_archiver.c:235
AH = 0xe6cf90
#5 0x0000000000403eff in main (argc=7, argv=0x7fffb0559018)
at pg_restore.c:400
opts = 0xe8b9e0
c = -1
exit_code = 32681
numWorkers = 1
AH = 0x7fa9b647667b <do_lookup_x+2011>
inputFileSpec = 0x0
disable_triggers = 0
enable_row_security = 0
if_exists = 0
no_data_for_failed_tables = 0
outputNoTablespaces = 0
use_setsessauth = 0
no_publications = 0
no_security_labels = 0
no_subscriptions = 0
strict_names = 0
cmdopts = {{name = 0x41edd8 "clean", has_arg = 0, flag = 0x0,
val = 99}, {name = 0x41edde "create", has_arg = 0, flag = 0x0,
val = 67}, {name = 0x41ede5 "data-only", has_arg = 0, flag =
0x0,
val = 97}, {name = 0x41edef "dbname", has_arg = 1, flag = 0x0,
val = 100}, {name = 0x41edf6 "exit-on-error", has_arg = 0,
flag = 0x0, val = 101}, {name = 0x41ee04 "exclude-schema",
has_arg = 1, flag = 0x0, val = 78}, {name = 0x41ee13 "file",
has_arg = 1, flag = 0x0, val = 102}, {name = 0x41ee18 "format",
has_arg = 1, flag = 0x0, val = 70}, {name = 0x41ee1f "function",
has_arg = 1, flag = 0x0, val = 80}, {name = 0x41ee28 "host",
has_arg = 1, flag = 0x0, val = 104}, {name = 0x41ee2d "index",
has_arg = 1, flag = 0x0, val = 73}, {name = 0x41ee33 "jobs",
has_arg = 1, flag = 0x0, val = 106}, {name = 0x41ee38 "list",
has_arg = 0, flag = 0x0, val = 108}, {
name = 0x41ee3d "no-privileges", has_arg = 0, flag = 0x0,
val = 120}, {name = 0x41ee4b "no-acl", has_arg = 0, flag = 0x0,
val = 120}, {name = 0x41ee52 "no-owner", has_arg = 0, flag =
0x0,
val = 79}, {name = 0x41ee5b "no-reconnect", has_arg = 0,
flag = 0x0, val = 82}, {name = 0x41ee68 "port", has_arg = 1,
flag = 0x0, val = 112}, {name = 0x41ee6d "no-password",
has_arg = 0, flag = 0x0, val = 119}, {name = 0x41ee79
"password",
has_arg = 0, flag = 0x0, val = 87}, {name = 0x41ee82 "schema",
has_arg = 1, flag = 0x0, val = 110}, {
name = 0x41ee89 "schema-only", has_arg = 0, flag = 0x0,
val = 115}, {name = 0x41ee95 "superuser", has_arg = 1, flag =
0x0,
val = 83}, {name = 0x41ee9f "table", has_arg = 1, flag = 0x0,
val = 116}, {name = 0x41eea5 "trigger", has_arg = 1, flag = 0x0,
val = 84}, {name = 0x41eead "use-list", has_arg = 1, flag = 0x0,
val = 76}, {name = 0x41eeb6 "username", has_arg = 1, flag = 0x0,
val = 85}, {name = 0x41eebf "verbose", has_arg = 0, flag = 0x0,
val = 118}, {name = 0x41eec7 "single-transaction", has_arg = 0,
flag = 0x0, val = 49}, {name = 0x41eeda "disable-triggers",
has_arg = 0, flag = 0x62c5ac <disable_triggers>, val = 1}, {
name = 0x41eeeb "enable-row-security", has_arg = 0,
flag = 0x62c5b0 <enable_row_security>, val = 1}, {
name = 0x41eeff "if-exists", has_arg = 0,
flag = 0x62c5cc <if_exists>, val = 1}, {
name = 0x41ef09 "no-data-for-failed-tables", has_arg = 0,
flag = 0x62c5b4 <no_data_for_failed_tables>, val = 1}, {
name = 0x41ef23 "no-tablespaces", has_arg = 0,
flag = 0x62c5b8 <outputNoTablespaces.7124>, val = 1}, {
name = 0x41ef32 "role", has_arg = 1, flag = 0x0, val = 2}, {
name = 0x41ef37 "section", has_arg = 1, flag = 0x0, val = 3},
{
name = 0x41ef3f "strict-names", has_arg = 0,
flag = 0x62c5d0 <strict_names>, val = 1}, {
name = 0x41ef4c "use-set-session-authorization", has_arg = 0,
flag = 0x62c5bc <use_setsessauth>, val = 1}, {
name = 0x41ef6a "no-publications", has_arg = 0,
flag = 0x62c5c0 <no_publications>, val = 1}, {
name = 0x41ef7a "no-security-labels", has_arg = 0,
flag = 0x62c5c4 <no_security_labels>, val = 1}, {
name = 0x41ef8d "no-subscriptions", has_arg = 0,
flag = 0x62c5c8 <no_subscriptions>, val = 1}, {name = 0x0,
has_arg = 0, flag = 0x0, val = 0}}
quit
Detaching from program:
/usr0/home/skoposov_remote/postgresql-10.16/src/bin/pg_dump/pg_restore,
process 3461
PG Bug reporting form <noreply@postgresql.org> writes:
I have a reproducible case of segfaulting pg_restore when trying to restore
from pg_dump of a different version. Specifically at least pg_restore from
10 crashes from pg_dump 12.
When I try that I get
pg_restore: [archiver] unsupported version (1.14) in file header
and that test is done first thing in ReadHead(), before the place
you identify. I suspect you are dealing with a corrupt archive
file, not a version mismatch.
regards, tom lane
On Wed, 2021-03-31 at 17:27 -0400, Tom Lane wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.PG Bug reporting form <noreply@postgresql.org> writes:
I have a reproducible case of segfaulting pg_restore when trying to restore
from pg_dump of a different version. Specifically at least pg_restore from
10 crashes from pg_dump 12.When I try that I get
pg_restore: [archiver] unsupported version (1.14) in file header
and that test is done first thing in ReadHead(), before the place
you identify. I suspect you are dealing with a corrupt archive
file, not a version mismatch.
I am pretty sure that is not the case.
I've just done that now
skoposov_remote@wsdb:~/postgresql-10.16$ ssh -o Compression=no koposov@HOSTNAME '/opt/pgsql/bin/pg_dump -n SCHEMA -Fc -U dbadmin wsdb' | head -c 32768 >
xx.short
(obviously the crash doesn't happen even if I don't put 'head -n ...')
skoposov_remote@wsdb:~/postgresql-10.16$ cat xx.short | pg_restore -U dbadmin -h localhost -d wsdb
Segmentation fault (core dumped)
I attach the file (1 kb of it)
I also noticed when I tried to run pg_restore in the debugger the crash doesn't happen.
gdb --args pg_restore -U dbadmin -h localhost -d wsdb
(gdb) run < /tmp/xx.short
Starting program: /usr0/home/skoposov_remote/postgresql-10.16/src/bin/pg_dump/pg_restore < /tmp/xx.short
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
pg_restore: [archiver] unsupported version (1.14) in file header
[Inferior 1 (process 14109) exited with code 01]
but it does if I pipe it...
Sergey
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
Attachments:
Sergey KOPOSOV <Sergey.Koposov@ed.ac.uk> writes:
I also noticed when I tried to run pg_restore in the debugger the crash doesn't happen.
but it does if I pipe it...
I wonder if your platform is helpfully inserting Windows newlines,
or perhaps removing them, when the data goes through a pipe.
regards, tom lane
On Wed, 2021-03-31 at 18:47 -0400, Tom Lane wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.Sergey KOPOSOV <Sergey.Koposov@ed.ac.uk> writes:
I also noticed when I tried to run pg_restore in the debugger the crash doesn't happen.
but it does if I pipe it...I wonder if your platform is helpfully inserting Windows newlines,
or perhaps removing them, when the data goes through a pipe.
The platform doing the pg_dump (PG12) is debian
And the one doing pg_restore (PG10) is ubuntu.
Also I know for sure when I use pg_restore from postgresql 12 it works fine in the same configuration.
(And I've regularly transferred tables this way from one system to another, it's just I've recently
migrated the debian system from PG11 to PG12 which lead to this segfault.)
S
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
On Wed, 2021-03-31 at 23:52 +0100, Sergey Koposov wrote:
On Wed, 2021-03-31 at 18:47 -0400, Tom Lane wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.Sergey KOPOSOV <Sergey.Koposov@ed.ac.uk> writes:
I also noticed when I tried to run pg_restore in the debugger the crash doesn't happen.
but it does if I pipe it...I wonder if your platform is helpfully inserting Windows newlines,
or perhaps removing them, when the data goes through a pipe.The platform doing the pg_dump (PG12) is debian
And the one doing pg_restore (PG10) is ubuntu.Also I know for sure when I use pg_restore from postgresql 12 it works fine in the same configuration.
(And I've regularly transferred tables this way from one system to another, it's just I've recently
migrated the debian system from PG11 to PG12 which lead to this segfault.)
I've just verified that on a different machine ubuntu 18.04 I can crash pg_restore with the file that I've sent to the list.
Importantly This requires running pg_restore without '-Fc' flag, i.e. to let it autodetect.
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore
Segmentation fault (core dumped)
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore -Fc
pg_restore: [archiver] unsupported version (1.14) in file header
S
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
Sergey KOPOSOV <Sergey.Koposov@ed.ac.uk> writes:
Importantly This requires running pg_restore without '-Fc' flag, i.e. to let it autodetect.
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore
Segmentation fault (core dumped)
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore -Fc
pg_restore: [archiver] unsupported version (1.14) in file header
Ooooh ... the autodetect + cant-seek code path is just broken. All of the
sanity checks on the first few fields of the file --- particularly the
version number --- just get skipped in this scenario.
I wonder why it's a good idea to read-ahead any of those fields in the
first place. Checking the PGDMP magic string seems sufficient.
Will fix, thanks for the report!
regards, tom lane
On Thu, Apr 01, 2021 at 11:39:33AM -0400, Tom Lane wrote:
Sergey KOPOSOV <Sergey.Koposov@ed.ac.uk> writes:
Importantly This requires running pg_restore without '-Fc' flag, i.e. to let it autodetect.
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore
Segmentation fault (core dumped)
$ cat /tmp/xx1.short | ./src/bin/pg_dump/pg_restore -Fc
pg_restore: [archiver] unsupported version (1.14) in file headerOoooh ... the autodetect + cant-seek code path is just broken. All of the
sanity checks on the first few fields of the file --- particularly the
version number --- just get skipped in this scenario.I wonder why it's a good idea to read-ahead any of those fields in the
first place. Checking the PGDMP magic string seems sufficient.Will fix, thanks for the report!
Yes, thank you both. I've run into this recently but for some reason I thought
it was fixed. It probably also explains this one from 2014.
/messages/by-id/20141206061151.GA725@telsasoft.com
--
Justin
Justin Pryzby <pryzby@telsasoft.com> writes:
On Thu, Apr 01, 2021 at 11:39:33AM -0400, Tom Lane wrote:
Will fix, thanks for the report!
Yes, thank you both. I've run into this recently but for some reason I thought
it was fixed. It probably also explains this one from 2014.
/messages/by-id/20141206061151.GA725@telsasoft.com
Yeah, this does look suspiciously like it explains some past reports
that we failed to reproduce, likely because it didn't occur to us that
reading the file from a non-seekable source would make a difference.
Thanks to Sergey for beating me over the head till I didn't dismiss
it anymore ;-)
regards, tom lane