base backup and tar problems with disappearing files.

On 15 Mar 2011, at 3:06, Rajesh Kumar Mallah wrote:

Dear Friends,

While taking online basebackup we ignore tar exit codes of 1 .
However under certain circumstances tar exits we code '2' which
stands for 'Fatal Errors' . Eg in case of "Cannot stat: No such file or directory"
encountered while taking backup of the pgdatadir . My question is
can we ignore such errors of "vanishing files" ? is it normal ?
I think the situation is arising because some table which were created
before start of backup were dropped during the backup. But that is
quite normal also.

You should probably exclude the PG data directories from your file-system backups, there isn't much point in backing them up anyway.

Alban Hertroys

--
Screwing up is an excellent way to attach something to the ceiling.

!DSPAM:737,4d7f0b68235882060612259!

On 15 Mar 2011, at 7:46, Alban Hertroys wrote:

On 15 Mar 2011, at 3:06, Rajesh Kumar Mallah wrote:

Dear Friends,

While taking online basebackup we ignore tar exit codes of 1 .
However under certain circumstances tar exits we code '2' which
stands for 'Fatal Errors' . Eg in case of "Cannot stat: No such file or directory"
encountered while taking backup of the pgdatadir . My question is
can we ignore such errors of "vanishing files" ? is it normal ?
I think the situation is arising because some table which were created
before start of backup were dropped during the backup. But that is
quite normal also.

You should probably exclude the PG data directories from your file-system backups, there isn't much point in backing them up anyway.

I should refine that a bit...

A file-system level backup backs up the files in a sequential order, while the database writes it's transactions in them in a pattern that's much closer to random order. As a result of that, your file-system backup is likely to contain the database files in an inconsistent state.
If you subsequently try to recover from that backup, you rely on the ability of the database to recover from that inconsistent state. Postgres is pretty good at recovering, but there's no guarantee it will succeed. It's probably a bad idea to rely on that for your backups.

Instead, for backing up your database, use one of the strategies outlined in the fine manual. Those are reliable.

Alban Hertroys

--
If you can't see the forest for the trees,
cut the trees and you'll see there is no forest.

!DSPAM:737,4d7f1594235881055444518!

On 03/15/11 12:30 AM, Alban Hertroys wrote:

On 15 Mar 2011, at 7:46, Alban Hertroys wrote:

On 15 Mar 2011, at 3:06, Rajesh Kumar Mallah wrote:

Dear Friends,

While taking online basebackup we ignore tar exit codes of 1 .
However under certain circumstances tar exits we code '2' which
stands for 'Fatal Errors' . Eg in case of "Cannot stat: No such file or directory"
encountered while taking backup of the pgdatadir . My question is
can we ignore such errors of "vanishing files" ? is it normal ?
I think the situation is arising because some table which were created
before start of backup were dropped during the backup. But that is
quite normal also.

You should probably exclude the PG data directories from your file-system backups, there isn't much point in backing them up anyway.

I should refine that a bit...

A file-system level backup backs up the files in a sequential order, while the database writes it's transactions in them in a pattern that's much closer to random order. As a result of that, your file-system backup is likely to contain the database files in an inconsistent state.
If you subsequently try to recover from that backup, you rely on the ability of the database to recover from that inconsistent state. Postgres is pretty good at recovering, but there's no guarantee it will succeed. It's probably a bad idea to rely on that for your backups.

Instead, for backing up your database, use one of the strategies outlined in the fine manual. Those are reliable.

um, I assumed from the original post that he was talking about taking a
base backup in preparation for setting up WAL replication, presumably
preceded by a call to pg_start_backup(), etc...

Sorry Alban , For the late update.

Yep John,

I am talking about the tar'ring of pgdatadir only excluding the pg_xlog
dir.
We have set up our full backup system in accordance to the admin guide.
Even the guide puts forward the limitation of tar in producing
distinguishing exit codes.

My doubt at this moment is , Is it normal to expect files disappearing from
the
pgdatadir during the course of taking base backup ? I can think about the
temp
sorting files disappearing but i am not sure what could cause data files
disappear
like example given below:

tar: /mnt/disk1/pgdatadir/base/16399/861272781: Cannot stat: No such file or
directory

If the above incidence is normal then i would only worry about making tar
not to worry
about the disappearing files.

The reason of putting this question is that the line in one of the later
paras
of section "24.3.2" says

"Some file system backup tools emit warnings or errors if the files they are
trying to
copy change while the copy proceeds." , it only says about "change" not
disappearance.

Since i have deep respect for the excellent documentation quality of
PostgreSQL project
i read and interpret it by words.

Warm Regds
-mallah.

On Tue, Mar 15, 2011 at 1:42 PM, John R Pierce <pierce@hogranch.com> wrote: