consistent postgresql snapshot

On 5/11/22 10:41, Zwettler Markus (OIZ) wrote:

PostgreSQL12 running on CentOS7 + ext4.

Would it be save to do a "fsfreeze –freeze" + disk snapshot + "fsfreeze
–unfreeze" of the PGDATA filesystem to get a consistent snapshot?

I am wondering as PostgreSQL default blocksize = 8k while ext4 default
blocksize = 4k, resulting in 2 fs blocks per db block.

If data/base and pg_xlog are on the same filesystem, then it might work.

Be sure to select pg_start_backup('something informative'); beforehand, and
select pg_stop_backup(); afterwards.

For example:
psql -c "select pg_start_backup('something informative');"
fsfreeze –freeze
/disk snapshot/
fsfreeze –unfreeze
psql -c "select pg_stop_backup();"

--
Angular momentum makes the world go 'round.

I don't want to do use the normal backup algorithm where pg_start_backup + pg_stop_backup will fix any fractured block and I am required to have all archived logfiles, therefore.

I want to produce an atomic consistent disk snapshot.

I am not sure if freeze/unfreeze will be enough with default values (8k db blocksize vs. 4k fs blocksize).

I am quite sure it should be safe when I also change the fs blocksize to 8k because 1 IO = 1 db block = 1 fs block than.

Any thoughts?

Von: Ron <ronljohnsonjr@gmail.com>
Gesendet: Donnerstag, 12. Mai 2022 01:51
An: pgsql-general@lists.postgresql.org
Betreff: [Extern] Re: consistent postgresql snapshot

On 5/11/22 10:41, Zwettler Markus (OIZ) wrote:

PostgreSQL12 running on CentOS7 + ext4.

Would it be save to do a "fsfreeze –freeze" + disk snapshot + "fsfreeze –unfreeze" of the PGDATA filesystem to get a consistent snapshot?

I am wondering as PostgreSQL default blocksize = 8k while ext4 default blocksize = 4k, resulting in 2 fs blocks per db block.

If data/base and pg_xlog are on the same filesystem, then it might work.

Be sure to select pg_start_backup('something informative'); beforehand, and select pg_stop_backup(); afterwards.

For example:
psql -c "select pg_start_backup('something informative');"
fsfreeze –freeze
disk snapshot
fsfreeze –unfreeze
psql -c "select pg_stop_backup();"
--
Angular momentum makes the world go 'round.

Achtung: Diese E-Mail wurde von einer externen Adresse verschickt. Klicken Sie auf keine Links und öffnen Sie keine angehängten Dateien, wenn Sie den Absender bzw. die Absenderin nicht kennen. Sind Sie sich unsicher, kontaktieren Sie den Service Desk der Stadt Zürich.

"Zwettler Markus (OIZ)" <Markus.Zwettler@zuerich.ch> writes:

I don't want to do use the normal backup algorithm where pg_start_backup + pg_stop_backup will fix any fractured block and I am required to have all archived logfiles, therefore.
I want to produce an atomic consistent disk snapshot.

[ shrug... ] You can't have that. There is never any guarantee that
the on-disk database files are fully up-to-date while the Postgres
server is running, because there may be updates sitting in buffers
in shared memory that have not been written out yet. We achieve
crash safety by ensuring that the WAL log contains sufficient info to
recreate any such updates by replaying WAL from the last checkpoint.
But without going through that replay process, the data visible in
the filesystem may be inconsistent. Even taking a disk snapshot
immediately after a checkpoint won't help, because we use spread
checkpoints. These points are independent of the possibility that
any one block write is non-atomic, although that's surely a factor
as well.

The only way you could get a consistent on-disk image is to shut
the server down (being sure to do a clean not "immediate" shutdown)
and then take the snapshot.

regards, tom lane

On Thu, 12 May 2022 at 14:48, Tom Lane <tgl@sss.pgh.pa.us> wrote:

"Zwettler Markus (OIZ)" <Markus.Zwettler@zuerich.ch> writes:

I don't want to do use the normal backup algorithm where pg_start_backup

+ pg_stop_backup will fix any fractured block and I am required to have all
archived logfiles, therefore.

I want to produce an atomic consistent disk snapshot.

[ shrug... ] You can't have that. [snip]

The only way you could get a consistent on-disk image is to shut
the server down (being sure to do a clean not "immediate" shutdown)
and then take the snapshot.

I think you could work around that by taking a dirty snapshot, making a
writable filesystem from it, waiting until you've archived enough WAL to
get that to a consistent state, and then firing up a temporary postmaster
on that filesystem to go through recovery and shut down cleanly.

Talking about fsfreeze and blocksize are not relevant in your case at all.
You can't make a backup this way any way. According your mail,
you are playing with database recovery after crash. Is pg crash proof? Yes (
https://www.postgresql.org/docs/current/wal-intro.html).
You can use this solution for example to make a test environment and it
works,
but not for live database backup.
For backup use a pg_basebackup or pg_start_backup()/snap/pg_stop_backup()
solution
br
Kaido

On Thu, 12 May 2022 at 17:53, Nick Cleaton <nick@cleaton.net> wrote:

Whether or not you want to use the "normal backup algorithm" is irrelevant,
since if you try and start that snapshot Postgresql /*will*/ see the
instance as /*needing*/ to be recovered.

Having said that... our storage team does SAN snapshots of the *single LUN*
where */all/**of the Postgres* files (regardless of mountpoint) reside on
"my" Postgresql server.  When needed, the Linux guys restored the relevant
snapshot to a new server, changed the host name and IP address and Postgres
started right up, recovering just as if there had been a system crash.

On 5/12/22 04:10, Zwettler Markus (OIZ) wrote:

Email Signature

I don't want to do use the normal backup algorithm where pg_start_backup +
pg_stop_backup will fix any fractured block and I am required to have all
archived logfiles, therefore.

I want to produce an atomic consistent disk snapshot.

I am not sure if freeze/unfreeze will be enough with default values (8k db
blocksize vs. 4k fs blocksize).

I am quite sure it should be safe when I also change the fs blocksize to
8k because 1 IO = 1 db block = 1 fs block than.

Any thoughts?

*Von:*Ron <ronljohnsonjr@gmail.com>
*Gesendet:* Donnerstag, 12. Mai 2022 01:51
*An:* pgsql-general@lists.postgresql.org
*Betreff:* [Extern] Re: consistent postgresql snapshot

On 5/11/22 10:41, Zwettler Markus (OIZ) wrote:

PostgreSQL12 running on CentOS7 + ext4.

Would it be save to do a "fsfreeze –freeze" + disk snapshot +
"fsfreeze –unfreeze" of the PGDATA filesystem to get a consistent
snapshot?

I am wondering as PostgreSQL default blocksize = 8k while ext4 default
blocksize = 4k, resulting in 2 fs blocks per db block.

If data/base and pg_xlog are on the same filesystem, then it might work.

Be sure to select pg_start_backup('something informative');beforehand, and
select pg_stop_backup();afterwards.

For example:
psql -c "select pg_start_backup('something informative');"
fsfreeze –freeze
/disk snapshot/
fsfreeze –unfreeze
psql -c "select pg_stop_backup();"

--
Angular momentum makes the world go 'round.

Achtung: Diese E-Mail wurde von einer externen Adresse verschickt. Klicken
Sie auf keine Links und öffnen Sie keine angehängten Dateien, wenn Sie den
Absender bzw. die Absenderin nicht kennen. Sind Sie sich unsicher,
kontaktieren Sie den Service Desk der Stadt Zürich.

--
Angular momentum makes the world go 'round.