PG backup check

On Mon, Jul 16, 2018 at 8:35 PM, Ravi Krishna <sravikrishna3@gmail.com>
wrote:

Not sure I am following this. Did Google release this because PG backups
are not 100% reliable or the data corruption can occur due to hardware
failure.

http://www.eweek.com/cloud/google-releases-open-source-
tool-that-checks-postgres-backup-integrity?utm_medium=
email&utm_campaign=EWK_NL_EP_20180713_STR5L2&dni=450493554&rni=24844166

The rule of thumb is you should verify that your backups can be restored
before you consider them reliable. You don't want to end up in a disaster
recovery situation and find that your backups have been corrupted in any
way shape or form. Many things can break your backups, especially if they
are automated and have many steps involved.

One way to do this is to actually restore the data on a different system
and compare. This tool appears to do the comparison for you directly to the
dump, increasing confidence that it is reliable.

2018-07-17 2:35 GMT+02:00 Ravi Krishna <sravikrishna3@gmail.com>:

Not sure I am following this. Did Google release this because PG backups
are not 100% reliable or the data corruption can occur due to hardware
failure.

http://www.eweek.com/cloud/google-releases-open-source-
tool-that-checks-postgres-backup-integrity?utm_medium=
email&utm_campaign=EWK_NL_EP_20180713_STR5L2&dni=450493554&rni=24844166

From what I understand with this Google tool, it has nothing to do with
backups. It just allows you to check data blocks in a PostgreSQL cluster.
Google advice is to run it before taking a backup, but that's about it.

--
Guillaume.

On Tue, Jul 17, 2018 at 11:17 AM, Guillaume Lelarge <guillaume@lelarge.info>
wrote:

2018-07-17 2:35 GMT+02:00 Ravi Krishna <sravikrishna3@gmail.com>:

Not sure I am following this. Did Google release this because PG backups
are not 100% reliable or the data corruption can occur due to hardware
failure.

http://www.eweek.com/cloud/google-releases-open-source-tool-
that-checks-postgres-backup-integrity?utm_medium=email&
utm_campaign=EWK_NL_EP_20180713_STR5L2&dni=450493554&rni=24844166

From what I understand with this Google tool, it has nothing to do with
backups. It just allows you to check data blocks in a PostgreSQL cluster.
Google advice is to run it before taking a backup, but that's about it.

This appears to basically be the same tool that's already included in
PostgreSQL 11, and has been around in a few different incarnations (but
unpolished) for years.

FWIW, in relation to backups, tools like pgbackrest already did this
transparently during backup, and again PostgreSQL 11 will do it built-in.

It's quite possible Google was running this internally before of course,
and a separate tool from others, but it's not exactly news... But they do
outline a very definite problem, which is that if you get physical
corruption in your database, it gets included in the backups. And if it's
in a portion of the database you don't use a lot, checksum failures won't
be noticed until you actually try, which is way too late.

--
Magnus Hagander
Me: https://www.hagander.net/ <http://www.hagander.net/&gt;
Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/&gt;