power failure....

Started by Tom Allisonover 18 years ago10 messagesgeneral
Jump to latest
#1Tom Allison
tom@tacocat.net

How do I restart the following with some level of sanity?

2007-11-30 19:35:20 EST LOG: could not load root certificate file
"root.crt": no SSL error reported
2007-11-30 19:35:20 EST DETAIL: Will not verify client certificates.
2007-11-30 19:35:20 EST LOG: database system was interrupted at
2007-11-27 09:08:44 EST
2007-11-30 19:35:20 EST LOG: record with zero length at 0/40808BA0
2007-11-30 19:35:20 EST LOG: invalid primary checkpoint record
2007-11-30 19:35:20 EST LOG: record with zero length at 0/407C9628
2007-11-30 19:35:20 EST LOG: invalid secondary checkpoint record
2007-11-30 19:35:20 EST PANIC: could not locate a valid checkpoint
record
2007-11-30 19:35:20 EST LOG: startup process (PID 8755) was
terminated by signal 6
2007-11-30 19:35:20 EST LOG: aborting startup due to startup process
failure

postgres version 8.2 sitting on a Reiser FS on RAID1....

I'm not finding much on the web other than bad news...

#2Tom Lane
tgl@sss.pgh.pa.us
In reply to: Tom Allison (#1)
Re: power failure....

Tom Allison <tom@tacocat.net> writes:

2007-11-30 19:35:20 EST PANIC: could not locate a valid checkpoint
record

Ugh :-(. pg_resetxlog should get you back into the database, but it's
anybody's guess whether and how badly the contents will be corrupted.
I would recommend trying a dump/reload and also doing any manual
consistency crosschecks you can think of.

postgres version 8.2 sitting on a Reiser FS on RAID1....

Better take a second look at your disk hardware configuration. At least
in theory, this Can't Happen if your disk hardware is handling fsync
honestly.

regards, tom lane

#3Tom Allison
tom@tacocat.net
In reply to: Tom Lane (#2)
Re: power failure....

Tom Lane wrote:

Tom Allison <tom@tacocat.net> writes:

2007-11-30 19:35:20 EST PANIC: could not locate a valid checkpoint
record

Ugh :-(. pg_resetxlog should get you back into the database, but it's
anybody's guess whether and how badly the contents will be corrupted.
I would recommend trying a dump/reload and also doing any manual
consistency crosschecks you can think of.

postgres version 8.2 sitting on a Reiser FS on RAID1....

Better take a second look at your disk hardware configuration. At least
in theory, this Can't Happen if your disk hardware is handling fsync
honestly.

regards, tom lane

Cool, thanks.

I wonder what I'm supposed to do with my debian installation since there doesn't
seem to be any such thing as pg_resetxlog. Or is it hiding somewhere?

#4Tom Allison
tom@tacocat.net
In reply to: Tom Lane (#2)
Re: power failure....

Tom Lane wrote:

Tom Allison <tom@tacocat.net> writes:

2007-11-30 19:35:20 EST PANIC: could not locate a valid checkpoint
record

Ugh :-(. pg_resetxlog should get you back into the database, but it's
anybody's guess whether and how badly the contents will be corrupted.
I would recommend trying a dump/reload and also doing any manual
consistency crosschecks you can think of.

postgres version 8.2 sitting on a Reiser FS on RAID1....

Better take a second look at your disk hardware configuration. At least
in theory, this Can't Happen if your disk hardware is handling fsync
honestly.

regards, tom lane

I tracked it down and did a reset.
I only have one large table right now.

And now I've decided to start using fsync=on!!! :)
I don't think I want to "test" this just yet...

#5Joshua D. Drake
jd@commandprompt.com
In reply to: Tom Allison (#3)
Re: power failure....

On Sat, 01 Dec 2007 15:22:38 -0500
Tom Allison <tom@tacocat.net> wrote:
regards, tom lane

Cool, thanks.

I wonder what I'm supposed to do with my debian installation since
there doesn't seem to be any such thing as pg_resetxlog. Or is it
hiding somewhere?

I don't recall if it is in contrib or not.. try?:

apt-file is your friend:

apt-file search pg_resetxlog

/usr/lib/postgresql/8.1/bin/pg_resetxlog

Joshua D. Drake

Show quoted text

---------------------------(end of
broadcast)--------------------------- TIP 5: don't forget to increase
your free space map settings

#6Joshua D. Drake
jd@commandprompt.com
In reply to: Tom Allison (#4)
Re: power failure....

On Sat, 01 Dec 2007 15:32:50 -0500
Tom Allison <tom@tacocat.net> wrote:

I tracked it down and did a reset.
I only have one large table right now.

And now I've decided to start using fsync=on!!! :)

change wal_sync_method to open_sync and fsync=on isn't nearly as bad as
it sounds.

joshua drake

Show quoted text

I don't think I want to "test" this just yet...

---------------------------(end of
broadcast)--------------------------- TIP 6: explain analyze is your
friend

#7Martin Marques
martin@marquesminen.com.ar
In reply to: Tom Allison (#3)
Re: power failure....

Tom Allison escribi�:

I wonder what I'm supposed to do with my debian installation since there
doesn't seem to be any such thing as pg_resetxlog. Or is it hiding
somewhere?

On one debian I have:

/usr/lib/postgresql/8.1/bin/pg_resetxlog
/usr/lib/postgresql/8.2/bin/pg_resetxlog

#8Martin Marques
martin@marquesminen.com.ar
In reply to: Joshua D. Drake (#5)
Re: power failure....

Joshua D. Drake escribi�:

On Sat, 01 Dec 2007 15:22:38 -0500

I wonder what I'm supposed to do with my debian installation since
there doesn't seem to be any such thing as pg_resetxlog. Or is it
hiding somewhere?

I don't recall if it is in contrib or not.. try?:

apt-file is your friend:

apt-file search pg_resetxlog

/usr/lib/postgresql/8.1/bin/pg_resetxlog

$ dlocate /usr/lib/postgresql/8.2/bin/pg_resetxlog
postgresql-8.2: /usr/lib/postgresql/8.2/bin/pg_resetxlog

#9Greg Smith
gsmith@gregsmith.com
In reply to: Joshua D. Drake (#6)
Re: power failure....

On Sat, 1 Dec 2007, Joshua D. Drake wrote:

change wal_sync_method to open_sync and fsync=on isn't nearly as bad as
it sounds.

Just be warned that there's been one report that some Linux versions have
bugs that make open_sync problematic:

http://archives.postgresql.org/pgsql-hackers/2007-10/msg01310.php

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

#10Tom Allison
tom@tacocat.net
In reply to: Greg Smith (#9)
Re: power failure....

I was able to get things up and running OK.
Don't have any WAL that I'm aware of, but it managed to have another
power failure hours later.

I seems that the UPS is more POS than UPS. I think the battery is dead.

On Dec 2, 2007, at 3:52 AM, Greg Smith wrote:

Show quoted text

On Sat, 1 Dec 2007, Joshua D. Drake wrote:

change wal_sync_method to open_sync and fsync=on isn't nearly as
bad as
it sounds.

Just be warned that there's been one report that some Linux
versions have bugs that make open_sync problematic:

http://archives.postgresql.org/pgsql-hackers/2007-10/msg01310.php

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com
Baltimore, MD