pg_basebackup

On 12/22/19 1:56 AM, Daulat Ram wrote:

Hello,

I am taking pg_basebackup of primary db (docker container env.) using
the below command :

pg_basebackup -x -h vmzti -U replication -D
/etc/postgresql/9.5/main/pg_basebkp/basekp1224 -Ft -z -P

Password:

WARNING:� skipping special file "./postgresql.conf"

WARNING:� skipping special file "./postgresql.conf"

4048512/4048512 kB (100%), 1/1 tablespace

https://www.postgresql.org/docs/9.5/app-pgbasebackup.html

"The backup will include all files in the data directory and
tablespaces, including the configuration files and any additional files
placed in the directory by third parties. But only regular files and
directories are copied. Symbolic links (other than those used for
tablespaces) and special device files are skipped. (See Section 50.3 for
the precise details.)"

postgresql.conf is a symlink in the data directory so it is being skipped.

* Getting the above warning message.
* Also the backup has been completed with only the single file� �tar
-xvf base.tar.gz� as an output but there is no� tar -xvf
pg_wal.tar.gz file exist.
* If we decompress the file� �tar -xvf base.tar.gz� �, I do not see
the postmaster.opts file.

Do you have more then one instance of Postgres running?

If so are you certain which one pg_basebackup is being pointed at?

Note: We have the different path for the conf files :

data_directory = '/var/lib/postgresql/9.5/main'�������� # use data in
another directory # (change requires restart)

hba_file = '/etc/postgresql/9.5/main/pg_hba.conf'������ # host-based
authentication file # (change requires restart)

ident_file = '/etc/postgresql/9.5/main/pg_ident.conf'�� # ident
configuration file # (change requires restart)

postgres=# SHOW config_file;

���������� ������config_file

----------------------------------------------

/var/lib/postgresql/9.5/main/postgresql.conf

(1 row)

postgres=# show hba_file;

�������������� hba_file

--------------------------------------

/etc/postgresql/9.5/main/pg_hba.conf

(1 row)

postgres=#

**

*More details:*

postgres@4ed92bac84af:/var/lib/postgresql/9.5/main$ ls -ltr

total 128

-rw------- 1 postgres postgres�� 208 Sep 13� 2017 backup_label.old

drwx------ 8 postgres postgres� 4096 Sep 13� 2017 base

drwx------ 2 postgres postgres� 4096 Sep 13 �2017 pg_commit_ts

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_dynshmem

drwx------ 4 postgres postgres� 4096 Sep 13� 2017 pg_multixact

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_replslot

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_serial

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_snapshots

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_stat

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_stat_tmp

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_tblspc

drwx------ 2 postgres postgres� 4096 Sep 13� 2017 pg_twophase

-rw------- 1 postgres postgres���� 4 Sep 13� 2017 PG_VERSION

-rw------- 1 postgres postgres��� 88 Sep 13� 2017 postgresql.auto.conf

-rwxr-xr-x 1 postgres postgres 22116 Sep 13� 2017 postgresql.conf.orig

drwx------ 2 postgres postgres 12288 Sep 13� 2017 pg_log

lrwxrwxrwx 1 postgres postgres��� 38 Sep 13� 2017 recovery.conf ->
/etc/postgresql/9.5/main/recovery.conf

lrwxrwxrwx 1 postgres postgres��� 40 Sep 13� 2017 postgresql.conf ->
/etc/postgresql/9.5/main/postgresql.conf

-rw------- 1 postgres postgres��� 90 May 21� 2019 postmaster.pid

drwx------ 2 postgres postgres� 4096 May 21� 2019 pg_notify

-rw------- 1 postgres postgres��� 46 May 21� 2019 postmaster.opts

drwx------ 2 postgres postgres� 8192 Dec 11 08:10 global

drwx------ 2 postgres postgres� 4096 Dec 17 20:12 pg_clog

drwx------ 2 postgres postgres� 4096 Dec 17 20:16 pg_subtrans

drwxrwxrwx 2 postgres postgres� 4096 Dec 18 08:31 backup

drwx------ 3 postgres postgres� 4096 Dec 18 15:56 pg_xlog

drwx------ 4 postgres postgres� 4096 Dec 18 16:31 pg_logical

postgres@4ed92bac84af:/var/lib/postgresql/9.5/main$

ident.conf

Please advise.

Thanks,

--
Adrian Klaver
adrian.klaver@aklaver.com

thanks Adrian, what about the
postmaster.opts file, this file was also skipped in backup.

We have single cluster running on the VM.

Thanks.
On 22-Dec-2019 11:19 PM, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
On 12/22/19 1:56 AM, Daulat Ram wrote:

Hello,

I am taking pg_basebackup of primary db (docker container env.) using
the below command :

pg_basebackup -x -h vmzti -U replication -D
/etc/postgresql/9.5/main/pg_basebkp/basekp1224 -Ft -z -P

Password:

WARNING: skipping special file "./postgresql.conf"

WARNING: skipping special file "./postgresql.conf"

4048512/4048512 kB (100%), 1/1 tablespace

https://www.postgresql.org/docs/9.5/app-pgbasebackup.html

"The backup will include all files in the data directory and
tablespaces, including the configuration files and any additional files
placed in the directory by third parties. But only regular files and
directories are copied. Symbolic links (other than those used for
tablespaces) and special device files are skipped. (See Section 50.3 for
the precise details.)"

postgresql.conf is a symlink in the data directory so it is being skipped.

* Getting the above warning message.
* Also the backup has been completed with only the single file “tar
-xvf base.tar.gz” as an output but there is no tar -xvf
pg_wal.tar.gz file exist.
* If we decompress the file “tar -xvf base.tar.gz” , I do not see
the postmaster.opts file.

Do you have more then one instance of Postgres running?

If so are you certain which one pg_basebackup is being pointed at?

Note: We have the different path for the conf files :

data_directory = '/var/lib/postgresql/9.5/main' # use data in
another directory # (change requires restart)

hba_file = '/etc/postgresql/9.5/main/pg_hba.conf' # host-based
authentication file # (change requires restart)

ident_file = '/etc/postgresql/9.5/main/pg_ident.conf' # ident
configuration file # (change requires restart)

postgres=# SHOW config_file;

config_file

----------------------------------------------

/var/lib/postgresql/9.5/main/postgresql.conf

(1 row)

postgres=# show hba_file;

hba_file

--------------------------------------

/etc/postgresql/9.5/main/pg_hba.conf

(1 row)

postgres=#

**

*More details:*

postgres@4ed92bac84af:/var/lib/postgresql/9.5/main$ ls -ltr

total 128

-rw------- 1 postgres postgres 208 Sep 13 2017 backup_label.old

drwx------ 8 postgres postgres 4096 Sep 13 2017 base

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_commit_ts

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_dynshmem

drwx------ 4 postgres postgres 4096 Sep 13 2017 pg_multixact

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_replslot

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_serial

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_snapshots

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_stat

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_stat_tmp

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_tblspc

drwx------ 2 postgres postgres 4096 Sep 13 2017 pg_twophase

-rw------- 1 postgres postgres 4 Sep 13 2017 PG_VERSION

-rw------- 1 postgres postgres 88 Sep 13 2017 postgresql.auto.conf

-rwxr-xr-x 1 postgres postgres 22116 Sep 13 2017 postgresql.conf.orig

drwx------ 2 postgres postgres 12288 Sep 13 2017 pg_log

lrwxrwxrwx 1 postgres postgres 38 Sep 13 2017 recovery.conf ->
/etc/postgresql/9.5/main/recovery.conf

lrwxrwxrwx 1 postgres postgres 40 Sep 13 2017 postgresql.conf ->
/etc/postgresql/9.5/main/postgresql.conf

-rw------- 1 postgres postgres 90 May 21 2019 postmaster.pid

drwx------ 2 postgres postgres 4096 May 21 2019 pg_notify

-rw------- 1 postgres postgres 46 May 21 2019 postmaster.opts

drwx------ 2 postgres postgres 8192 Dec 11 08:10 global

drwx------ 2 postgres postgres 4096 Dec 17 20:12 pg_clog

drwx------ 2 postgres postgres 4096 Dec 17 20:16 pg_subtrans

drwxrwxrwx 2 postgres postgres 4096 Dec 18 08:31 backup

drwx------ 3 postgres postgres 4096 Dec 18 15:56 pg_xlog

drwx------ 4 postgres postgres 4096 Dec 18 16:31 pg_logical

postgres@4ed92bac84af:/var/lib/postgresql/9.5/main$

ident.conf

Please advise.

Thanks,

--
Adrian Klaver
adrian.klaver@aklaver.com

Hello.

At Mon, 23 Dec 2019 03:38:12 +0000, Daulat Ram <Daulat.Ram@exponential.com> wrote in

thanks Adrian, what about the
postmaster.opts file, this file was also skipped in backup.

The file is overwritten at startup so there's no point in having it in
a backup. Thus, it is deliberately excluded from a backup.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

On Mon, Dec 23, 2019 at 4:13 AM Kyotaro Horiguchi <horikyota.ntt@gmail.com>
wrote:

Hello.

At Mon, 23 Dec 2019 03:38:12 +0000, Daulat Ram <Daulat.Ram@exponential.com>
wrote in

thanks Adrian, what about the
postmaster.opts file, this file was also skipped in backup.

The file is overwritten at startup so there's no point in having it in
a backup. Thus, it is deliberately excluded from a backup.

That is true and sensible, but it does seem to be contradicted by the
documentation for 9.5. In later versions, they added "except certain
temporary files managed by PostgreSQL". That addition seems to just be a
correction of the docs, not something to reflect a change in actual
behavior, at least in regards to postmaster.opts.

Cheers,

Jeff