Problem with pg_upgrade 8.3 to 9.1.4 - clog missing?!
hi
I just upgraded test copy of database of our customer (~ 600GB of data).
upgrade went fine, no errors. but vacuumdb -azv ended with an error:
=> vacuumdb --all --analyze -p 6665
vacuumdb: vacuuming database "client_db"
vacuumdb: vacuuming database "pg_audit"
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming of database "postgres" failed: ERROR: could not access status of transaction 860626316
DETAIL: Could not open file "pg_clog/0334": No such file or directory.
I know we had these kind of errors before, but I thought they all got fixed.
What can I do to debug it more?
8.3 was 8.3.16.
8.3.16 is from rpms, 9.1.4 is from source compilation.
both have integer datatimes, and both are for 64bit linux.
pg_clog contains:
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A4
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A5
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A6
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A7
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A8
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04A9
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AA
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AB
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AC
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AD
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AE
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04AF
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B0
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B1
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B2
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B3
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B4
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B5
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B6
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B7
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B8
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04B9
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BA
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BB
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BC
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BD
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BE
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04BF
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C0
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C1
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C2
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C3
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C4
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C5
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C6
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C7
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C8
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04C9
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CA
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CB
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CC
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CD
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CE
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04CF
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04D0
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04D1
-rw------- 1 postgres postgres 262144 Jun 6 21:22 04D2
-rw------- 1 postgres postgres 196608 Jun 7 00:15 04D3
unfortunately i do not have the 8.3 data dir any more.
if the problem is only with postgres database - i can ignore it. but I'd rather
be sure what is the source of the issue.
Best regards,
depesz
--
The best thing about modern society is how easy it is to avoid contact with it.
http://depesz.com/
On Thu, Jun 07, 2012 at 09:16:49PM +0200, hubert depesz lubaczewski wrote:
hi
I just upgraded test copy of database of our customer (~ 600GB of data).
upgrade went fine, no errors. but vacuumdb -azv ended with an error:=> vacuumdb --all --analyze -p 6665
vacuumdb: vacuuming database "client_db"
vacuumdb: vacuuming database "pg_audit"
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming of database "postgres" failed: ERROR: could not access status of transaction 860626316
DETAIL: Could not open file "pg_clog/0334": No such file or directory.I know we had these kind of errors before, but I thought they all got fixed.
What can I do to debug it more?
8.3 was 8.3.16.
8.3.16 is from rpms, 9.1.4 is from source compilation.both have integer datatimes, and both are for 64bit linux.
I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I
would suggest it was because we don't copy visibility map files from
pre-9.1 because those were not crash-safe, but 8.3 didn't have
visibility map files (added in PG 8.4), so 8.3 should have been checking
the exact same transaction ids as 9.1.
Basically, I can't think of a cause.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote:
I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I
would suggest it was because we don't copy visibility map files from
pre-9.1 because those were not crash-safe, but 8.3 didn't have
visibility map files (added in PG 8.4), so 8.3 should have been checking
the exact same transaction ids as 9.1.Basically, I can't think of a cause.
yeah, neither can we. but - any other test - even using the same source
of $PGDATA went flawless. I'm tempted to assume it's just a random
glitch in matrix, and just go on.
Best regards,
depesz
--
The best thing about modern society is how easy it is to avoid contact with it.
http://depesz.com/
On Wed, Jun 13, 2012 at 08:43:23PM +0200, hubert depesz lubaczewski wrote:
On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote:
I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I
would suggest it was because we don't copy visibility map files from
pre-9.1 because those were not crash-safe, but 8.3 didn't have
visibility map files (added in PG 8.4), so 8.3 should have been checking
the exact same transaction ids as 9.1.Basically, I can't think of a cause.
yeah, neither can we. but - any other test - even using the same source
of $PGDATA went flawless. I'm tempted to assume it's just a random
glitch in matrix, and just go on.
Is the vacuumdb still repeatedly failing?
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
On Wed, Jun 13, 2012 at 02:55:41PM -0400, Bruce Momjian wrote:
On Wed, Jun 13, 2012 at 08:43:23PM +0200, hubert depesz lubaczewski wrote:
On Wed, Jun 13, 2012 at 02:32:21PM -0400, Bruce Momjian wrote:
I assume 8.3 vacuumdb did not generate these errors. If it was 8.4 I
would suggest it was because we don't copy visibility map files from
pre-9.1 because those were not crash-safe, but 8.3 didn't have
visibility map files (added in PG 8.4), so 8.3 should have been checking
the exact same transaction ids as 9.1.Basically, I can't think of a cause.
yeah, neither can we. but - any other test - even using the same source
of $PGDATA went flawless. I'm tempted to assume it's just a random
glitch in matrix, and just go on.Is the vacuumdb still repeatedly failing?
well - in this particular upgraded database - i think so. I did run it
couple of times, it failed all the times.
But this $PGDATA is not longer used - we did another tests, and no other
test showed problems - i.e. vacuum worked fine.
Best regards,
depesz