pg_start_backup: file has vanished from pg_subtrans/

Leo Baltus wrote:

Hi,

While backing up some postgresql-8.4.2 instances using
pg_start_backup()/pg_stop_backup()

8.4.2 was released in 2009, so you're missing all the bugfixes till July
2014 which is when the 8.4 branch was dropped out of support -- which
means you're also missing bugfixes from that point onwards. I think you
need to jump to 8.4.22 (the last one in that release) because this might
be a fixed bug; and you should consider moving to one of the supported
branches very soon, at least 9.0 but that one's going to lose support
soon too so I may suggest something even newer.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 08/07/2015 07:20 AM, Alvaro Herrera wrote:

Leo Baltus wrote:

Hi,

While backing up some postgresql-8.4.2 instances using
pg_start_backup()/pg_stop_backup()

8.4.2 was released in 2009, so you're missing all the bugfixes till July
2014 which is when the 8.4 branch was dropped out of support -- which
means you're also missing bugfixes from that point onwards. I think you
need to jump to 8.4.22 (the last one in that release) because this might
be a fixed bug; and you should consider moving to one of the supported
branches very soon, at least 9.0 but that one's going to lose support
soon too so I may suggest something even newer.

That being said, I thought:

file has vanished: "............./data/pg_subtrans/3A37"

would not be a bug but normal behavior.

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Adrian Klaver wrote:

On 08/07/2015 07:20 AM, Alvaro Herrera wrote:

Leo Baltus wrote:

Hi,

While backing up some postgresql-8.4.2 instances using
pg_start_backup()/pg_stop_backup()

8.4.2 was released in 2009, so you're missing all the bugfixes till July
2014 which is when the 8.4 branch was dropped out of support -- which
means you're also missing bugfixes from that point onwards. I think you
need to jump to 8.4.22 (the last one in that release) because this might
be a fixed bug; and you should consider moving to one of the supported
branches very soon, at least 9.0 but that one's going to lose support
soon too so I may suggest something even newer.

That being said, I thought:

file has vanished: "............./data/pg_subtrans/3A37"

would not be a bug but normal behavior.

Well, it's true that pg_subtrans files vanish during regular running.
WAL replay of the resulting backup *should* work fine regardless, and
this is just rsync being paranoid, but I'm not sure that it *actually*
works.

--
�lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On 8/7/15 7:01 AM, Leo Baltus wrote:

While backing up some postgresql-8.4.2 instances using
pg_start_backup()/pg_stop_backup()

We noticed that sometimes rsync complains:

file has vanished: "............./data/pg_subtrans/3A37"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1039) [sender=3.0.6

It's perfectly normal for Postgres to remove files during a backup. If
a table is dropped then the files storing that table will be deleted.
Files can also be removed from pg_subtrans once they are no longer needed.

We already have excluded pgsql_tmp/ pg_xlog/ and pg_stat_tmp/ so I was
tempted to exclude pg_subtrans/ as well. However some googling seems to
suggest that this might invalidate the backup:

I would recommend only excluding pg_xlog as recommended in the Postgres
docs.

https://groups.google.com/forum/#!msg/pgbarman/zQxg6pOXkZ8/AmTmVtUA3dEJ

I am not sure what to believe, either
- my backup was invalid and I have to take measures to prevent this
(and file a bug report)

Your backup should be good so long as you've captured all the WAL.

- I can ignore this *and* can safely exclude pg_trans/ from my backups.
- there is some other workaround...

Do *not* exclude pg_subtrans.

You may also want to look at backup software such as pgBackRest. It
understands that files can be removed during backup and does not emit
warnings when it happens (though info messages are still logged
depending on your log level).

--
-David
david@pgmasters.net

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Op 08/08/2015 om 14:24:00 -0400, schreef David Steele:

On 8/7/15 7:01 AM, Leo Baltus wrote:

While backing up some postgresql-8.4.2 instances using
pg_start_backup()/pg_stop_backup()

We noticed that sometimes rsync complains:

file has vanished: "............./data/pg_subtrans/3A37"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1039) [sender=3.0.6

It's perfectly normal for Postgres to remove files during a backup. If a
table is dropped then the files storing that table will be deleted. Files
can also be removed from pg_subtrans once they are no longer needed.

We already have excluded pgsql_tmp/ pg_xlog/ and pg_stat_tmp/ so I was
tempted to exclude pg_subtrans/ as well. However some googling seems to
suggest that this might invalidate the backup:

I would recommend only excluding pg_xlog as recommended in the Postgres
docs.

Check.

https://groups.google.com/forum/#!msg/pgbarman/zQxg6pOXkZ8/AmTmVtUA3dEJ

I am not sure what to believe, either
- my backup was invalid and I have to take measures to prevent this
(and file a bug report)

Your backup should be good so long as you've captured all the WAL.

- I can ignore this *and* can safely exclude pg_trans/ from my backups.
- there is some other workaround...

Do *not* exclude pg_subtrans.

Ok, for now I will ignore rsync's exit code 24.

You may also want to look at backup software such as pgBackRest. It
understands that files can be removed during backup and does not emit
warnings when it happens (though info messages are still logged depending on
your log level).

Thanks!

--
Leo Baltus

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general