pg_basebackup cannot compress to STDOUNT

On 5/8/20 12:14 PM, Support wrote:

Hi,

Despite of the --help saying that it's possible to gzip to STDOUT and
pipe it for another process
pg_basebackup fails saying that it's not possible to gzip to STDOUT.

1) Postgres version?

2) Command run?

3) Error reported?

Who to believe then?

--
Adrian Klaver
adrian.klaver@aklaver.com

On 5/8/2020 12:18 PM, Adrian Klaver wrote:

On 5/8/20 12:14 PM, Support wrote:

Hi,

Despite of the --help saying that it's possible to gzip to STDOUT and
pipe it for another process
pg_basebackup fails saying that it's not possible to gzip to STDOUT.

1) Postgres version?

2) Command run?

3) Error reported?

Who to believe then?

1) Postgres version?
12.2 selt compiled with
./configure --with-perl --enable-integer-datetimes --enable-depend
--with-pam --with-systemd --enable-nls --with-libxslt --with-libxml
--with-llvm --with-python --with-icu --with-gssapi --with-openssl

2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz -c
-p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

3) Error reported?
pg_basebackup: error: cannot stream write-ahead logs in tar mode to stdout
Try "pg_basebackup --help" for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Thanks!

On 5/8/20 12:31 PM, Support wrote:

On 5/8/2020 12:18 PM, Adrian Klaver wrote:

On 5/8/20 12:14 PM, Support wrote:

Hi,

Despite of the --help saying that it's possible to gzip to STDOUT and
pipe it for another process
pg_basebackup fails saying that it's not possible to gzip to STDOUT.

1) Postgres version?

2) Command run?

3) Error reported?

Who to believe then?

1) Postgres version?
12.2 selt compiled with
./configure --with-perl --enable-integer-datetimes --enable-depend
--with-pam --with-systemd --enable-nls --with-libxslt --with-libxml
--with-llvm --with-python --with-icu --with-gssapi --with-openssl

2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz -c
-p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

3) Error reported?
pg_basebackup: error: cannot stream write-ahead logs in tar mode to stdout

https://www.postgresql.org/docs/12/app-pgbasebackup.html
"t
tar

Write the output as tar files in the target directory. The main
data directory will be written to a file named base.tar, and all other
tablespaces will be named after the tablespace OID.

If the value - (dash) is specified as target directory, the tar
contents will be written to standard output, suitable for piping to for
example gzip. This is only possible if the cluster has no additional
tablespaces and WAL streaming is not used.
^^^^^^^^^^^^^^^^^^^^^^^^^^
"

So use -X fetch or X none

Try "pg_basebackup --help" for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Thanks!

--
Adrian Klaver
adrian.klaver@aklaver.com

On 5/8/2020 1:24 PM, Adrian Klaver wrote:

On 5/8/20 12:31 PM, Support wrote:

On 5/8/2020 12:18 PM, Adrian Klaver wrote:

On 5/8/20 12:14 PM, Support wrote:

Hi,

Despite of the --help saying that it's possible to gzip to STDOUT
and pipe it for another process
pg_basebackup fails saying that it's not possible to gzip to STDOUT.

1) Postgres version?

2) Command run?

3) Error reported?

Who to believe then?

1) Postgres version?
12.2 selt compiled with
./configure --with-perl --enable-integer-datetimes --enable-depend
--with-pam --with-systemd --enable-nls --with-libxslt --with-libxml
--with-llvm --with-python --with-icu --with-gssapi --with-openssl

2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz
-c -p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

3) Error reported?
pg_basebackup: error: cannot stream write-ahead logs in tar mode to
stdout

https://www.postgresql.org/docs/12/app-pgbasebackup.html
"t
tar

    Write the output as tar files in the target directory. The main
data directory will be written to a file named base.tar, and all other
tablespaces will be named after the tablespace OID.

    If the value - (dash) is specified as target directory, the tar
contents will be written to standard output, suitable for piping to
for example gzip. This is only possible if the cluster has no
additional tablespaces and WAL streaming is not used.
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
"

So use -X fetch or X none

Try "pg_basebackup --help" or more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Thanks!

Good catch thank you!

On 5/8/20 1:42 PM, Support wrote:

On 5/8/2020 1:24 PM, Adrian Klaver wrote:

On 5/8/20 12:31 PM, Support wrote:

On 5/8/2020 12:18 PM, Adrian Klaver wrote:

On 5/8/20 12:14 PM, Support wrote:

Hi,

Despite of the --help saying that it's possible to gzip to STDOUT
and pipe it for another process
pg_basebackup fails saying that it's not possible to gzip to STDOUT.

1) Postgres version?

2) Command run?

3) Error reported?

Who to believe then?

1) Postgres version?
12.2 selt compiled with
./configure --with-perl --enable-integer-datetimes --enable-depend
--with-pam --with-systemd --enable-nls --with-libxslt --with-libxml
--with-llvm --with-python --with-icu --with-gssapi --with-openssl

2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz
-c -p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

3) Error reported?
pg_basebackup: error: cannot stream write-ahead logs in tar mode to
stdout

https://www.postgresql.org/docs/12/app-pgbasebackup.html
"t
tar

    Write the output as tar files in the target directory. The main
data directory will be written to a file named base.tar, and all other
tablespaces will be named after the tablespace OID.

    If the value - (dash) is specified as target directory, the tar
contents will be written to standard output, suitable for piping to
for example gzip. This is only possible if the cluster has no
additional tablespaces and WAL streaming is not used.
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
"

So use -X fetch or X none

Try "pg_basebackup --help" or more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Thanks!

Good catch thank you!

It was not much of a catch, the error message supplied the info:

"pg_basebackup: error: cannot stream write-ahead logs in tar mode to
stdout "

--
Adrian Klaver
adrian.klaver@aklaver.com

Hi Admin,

On 08. May, 2020, at 21:31, Support <admin@e-blokos.com> wrote:
2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz -c -p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

I don't get it, sorry. Do I understand you correctly here that you want an online backup or a *remotely* running PostgreSQL instance on your local machine?

If so, why not just let pg_basebackup connect remotely and let it do its magic? Something like this:

$ mkdir -p /usr/local/pgsql/data
$ cd /usr/local/pgsql/data
$ pg_basebackup -D /run/postgresql -Fp -P -v -h nodeXXX -p 5432 -U replicator
$ pg_ctl start

You'd have to have a role with replication privs or superuser and you'd have to adapt the port of course.

No need to take care of any WALs manually. It is all taken care of by pg_basebackup. The only real drawback is that if you have tablespaces, you'd have to create all directories of the tablespaces beforehand, which is why we removed them again after initially having tried the feature.

That's basically, how I create async replicas on out site, which is why I additionally add -R to the above command.

Cheers,
Paul

On 5/8/2020 11:51 PM, Paul Förster wrote:

Hi Admin,

On 08. May, 2020, at 21:31, Support <admin@e-blokos.com> wrote:
2) Command run?
ssh postgres@nodeXXX "pg_basebackup -h /run/postgresql -Ft -D- | pigz -c -p2 " | pigz -cd -p2 | tar -xf- -C /usr/local/pgsql/data

I don't get it, sorry. Do I understand you correctly here that you want an online backup or a *remotely* running PostgreSQL instance on your local machine?

If so, why not just let pg_basebackup connect remotely and let it do its magic? Something like this:

$ mkdir -p /usr/local/pgsql/data
$ cd /usr/local/pgsql/data
$ pg_basebackup -D /run/postgresql -Fp -P -v -h nodeXXX -p 5432 -U replicator
$ pg_ctl start

You'd have to have a role with replication privs or superuser and you'd have to adapt the port of course.

No need to take care of any WALs manually. It is all taken care of by pg_basebackup. The only real drawback is that if you have tablespaces, you'd have to create all directories of the tablespaces beforehand, which is why we removed them again after initially having tried the feature.

That's basically, how I create async replicas on out site, which is why I additionally add -R to the above command.

Cheers,
Paul

The trick of my command above is to get the transfer faster in one
compressed file going through the network.