We've been affected by a pg_upgrade bug. What do we do next?

Started by Shaun Thomasover 11 years ago10 messagesgeneral

sthomas@optionshouse.com

over 11 years ago

Hey,

We performed an upgrade via pg_upgrade from 9.1 to 9.3 a while back, and I'm almost certain we were bitten by this bug:

/messages/by-id/20140530121631.GE25431@alap3.anarazel.de

Finding the discussion is nice... but what do we do to fix this? I read through the discussion, and it *seems* we can delete the 0000 file and restart since it's only an 8k file and we haven't gone far enough to wrap into a new 0000 file. Will that actually work, though? Or is it too late, since something has already requested that invalid transaction? What do we do?

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

adrian.klaver@aklaver.com

over 11 years ago

In reply to: Shaun Thomas (#1)

Re: We've been affected by a pg_upgrade bug. What do we do next?

On 07/16/2014 07:08 AM, Shaun Thomas wrote:

Hey,

We performed an upgrade via pg_upgrade from 9.1 to 9.3 a while back, and I'm almost certain we were bitten by this bug:

/messages/by-id/20140530121631.GE25431@alap3.anarazel.de

Finding the discussion is nice... but what do we do to fix this? I read through the discussion, and it *seems* we can delete the 0000 file and restart since it's only an 8k file and we haven't gone far enough to wrap into a new 0000 file. Will that actually work, though? Or is it too late, since something has already requested that invalid transaction? What do we do?

See here:

https://wiki.postgresql.org/wiki/20140702pg_upgrade_fix

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Adrian Klaver
adrian.klaver@aklaver.com

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

sthomas@optionshouse.com

over 11 years ago

In reply to: Adrian Klaver (#2)

Re: We've been affected by a pg_upgrade bug. What do we do next?

See here:

https://wiki.postgresql.org/wiki/20140702pg_upgrade_fix

Thank $Deity! We'll schedule this ASAP. Our cluster is whining about this about every 10 seconds, which is kind of a PITA.

Thanks again!

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

andres@anarazel.de

over 11 years ago

In reply to: Shaun Thomas (#3)

Re: We've been affected by a pg_upgrade bug. What do we do next?

On 2014-07-16 14:24:12 +0000, Shaun Thomas wrote:

See here:

https://wiki.postgresql.org/wiki/20140702pg_upgrade_fix

Thank $Deity! We'll schedule this ASAP. Our cluster is whining about this about every 10 seconds, which is kind of a PITA.

What exactly is it whining about?

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

sthomas@optionshouse.com

over 11 years ago

In reply to: Andres Freund (#4)

Re: We've been affected by a pg_upgrade bug. What do we do next?

This:

23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|ERROR: could not access status of transaction 7150346
23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|DETAIL: Could not open file "pg_multixact/offsets/006D": No such file or directory.

Been doing it every ten seconds since yesterday.

My log line prefix is this:

log_line_prefix = '%p|%x|%u|%d|%r|%c|%t|'

So the user, database, and connection source are all blank. The PID is different every time, too. Not sure what to make of that.

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

alvherre@2ndquadrant.com

over 11 years ago

In reply to: Shaun Thomas (#5)

Re: We've been affected by a pg_upgrade bug. What do we do next?

Shaun Thomas wrote:

This:

23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|ERROR: could not access status of transaction 7150346
23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|DETAIL: Could not open file "pg_multixact/offsets/006D": No such file or directory.

Been doing it every ten seconds since yesterday.

My log line prefix is this:

log_line_prefix = '%p|%x|%u|%d|%r|%c|%t|'

So the user, database, and connection source are all blank. The PID is different every time, too. Not sure what to make of that.

It's an autovacuum worker, which is expected. Just get rid of the 0000
file and all should be well.

--
ï¿½lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

sthomas@optionshouse.com

over 11 years ago

In reply to: Alvaro Herrera (#6)

Re: We've been affected by a pg_upgrade bug. What do we do next?

It's an autovacuum worker, which is expected. Just get rid of the 0000
file and all should be well.

That's what I figured, but I didn't want to make assumptions. Does removing the 0000 file require a restart?

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

alvherre@2ndquadrant.com

over 11 years ago

In reply to: Alvaro Herrera (#6)

Re: We've been affected by a pg_upgrade bug. What do we do next?

Alvaro Herrera wrote:

Shaun Thomas wrote:

This:

23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|ERROR: could not access status of transaction 7150346
23334|4293964274||||53c68b69.5b26|2014-07-16 09:25:45 CDT|DETAIL: Could not open file "pg_multixact/offsets/006D": No such file or directory.

Been doing it every ten seconds since yesterday.

My log line prefix is this:

log_line_prefix = '%p|%x|%u|%d|%r|%c|%t|'

So the user, database, and connection source are all blank. The PID is different every time, too. Not sure what to make of that.

It's an autovacuum worker, which is expected. Just get rid of the 0000
file and all should be well.

BTW if you do a hexdump of the 0000 file, it should be all zeroes.

--
ï¿½lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

alvherre@2ndquadrant.com

over 11 years ago

In reply to: Shaun Thomas (#7)

Re: We've been affected by a pg_upgrade bug. What do we do next?

Shaun Thomas wrote:

It's an autovacuum worker, which is expected. Just get rid of the 0000
file and all should be well.

That's what I figured, but I didn't want to make assumptions. Does
removing the 0000 file require a restart?

Don't think so, but TBH I didn't try.

--
ï¿½lvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

sthomas@optionshouse.com

over 11 years ago

In reply to: Alvaro Herrera (#8)

Re: We've been affected by a pg_upgrade bug. What do we do next?

BTW if you do a hexdump of the 0000 file, it should be all zeroes.

Yep, that's what I get. :)

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general