Temporary tables prevent autovacuum, leading to XID wraparound

Started by Tsunakawa, Takayukiabout 8 years ago62 messageshackers
Jump to latest
#1Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com

Hello,

I've found a problem that an orphaned temporary table could cause XID wraparound. Our customer encountered this problem with PG 9.5.2, but I think this will happen with the latest PG.

I'm willing to fix this, but I'd like to ask you what approach we should take.

PROBLEM
====================================

The customer has a database for application data, which I call it user_db here. They don't store application data in postgres database.

No tables in user_db was autovacuumed for more than a month, leading to user tables bloating. Those tables are eligible for autovacuum according to pg_stat_all_tables and autovacuum settings.

age(datfrozenxid) of user_db and postgres databases are greater than autovacuum_max_freeze_age, so they are eligible for autovacuuming for XID wraparound.

There are user tables in user_db whose age(relfrozenxid) is greater than autovacuum_freeze_max_age, so those tables should get autovacuum treatment.

CAUSE
====================================

postgres database has a table named pg_temp_3.fetchchunks, whose age(relfrozenxid) is greater than autovacuum_freeze_max_age. This temporary table is the culprit. pg_temp_3.fetchchunks is created by pg_rewind. The customer says they ran pg_rewind.

autovacuum launcher always choose postgres, because do_start_worker() scans pg_database and finds that postgres database needs vacuuming for XID wraparound. user_db is never chosen for vacuuming, although it also needs vacuuming for XID wraparound.

autovacuum worker doesn't delete pg_temp3.fetchchunks, because the backendid 3 is used by some application so autovacuum worker thinks that the backend is active and the temporary table cannot be dropped.

I don't know why pg_temp3.fetchchunks still exists. Maybe the user ran pg_ctl stop -mi while pg_rewind was running.

FIX
====================================

I have the following questions. Along which line should I proceed to fix the problem?

* Why does autovacuum launcher always choose only one database when that database need vacuuming for XID wraparound? Shouldn't it also choose other databases?

* I think temporary tables should not require vacuuming for XID wraparound. Furtherover, should updates/deletes to temporary tables be in-place instead of creating garbage, so that any form of vacuum is unnecessary? Other sessions do not need to read temporary tables.

* In this incident, autovacuum worker misjudged that pg_temp_3.fetchchunks can't be deleted, although the creator (pg_rewind) is no longer active. How can we delete orphan temporary tables safely?

Regards
Takayuki Tsunakawa

#2Michael Paquier
michael@paquier.xyz
In reply to: Tsunakawa, Takayuki (#1)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Thu, Jan 25, 2018 at 06:14:41AM +0000, Tsunakawa, Takayuki wrote:

I don't know why pg_temp3.fetchchunks still exists. Maybe the user
ran pg_ctl stop -mi while pg_rewind was running.

Likely that was the case :(

As a superuser, DROP TABLE should work on the temporary schema of
another session. Have you tried that to solve the situation?

* I think temporary tables should not require vacuuming for XID
wraparound. Furtherover, should updates/deletes to temporary tables
be in-place instead of creating garbage, so that any form of vacuum is
unnecessary? Other sessions do not need to read temporary tables.

Yeah, there are many areas of improvements in this area. Temp tables
also generate WAL..

* In this incident, autovacuum worker misjudged that
pg_temp_3.fetchchunks can't be deleted, although the creator
(pg_rewind) is no longer active. How can we delete orphan temporary
tables safely?

As long as Postgres sees that its temporary schema is in use, it would
think that the table is not orphaned. Another thing possible would be to
have the session now holding this schema space to reuse fetchchunks so
as things are reset.
--
Michael

#3Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#2)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Michael Paquier [mailto:michael.paquier@gmail.com]
As a superuser, DROP TABLE should work on the temporary schema of another
session. Have you tried that to solve the situation?

Yes, we asked the customer to do that today. I think the customer will do in the near future.

* In this incident, autovacuum worker misjudged that
pg_temp_3.fetchchunks can't be deleted, although the creator
(pg_rewind) is no longer active. How can we delete orphan temporary
tables safely?

As long as Postgres sees that its temporary schema is in use, it would think
that the table is not orphaned. Another thing possible would be to have
the session now holding this schema space to reuse fetchchunks so as things
are reset.

I understood you suggested a new session which recycle the temp schema should erase the zombie metadata of old temp tables or recreate the temp schema. That sounds easy.

Regards
Takayuki Tsunakawa

#4Michael Paquier
michael@paquier.xyz
In reply to: Tsunakawa, Takayuki (#3)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Thu, Jan 25, 2018 at 08:10:00AM +0000, Tsunakawa, Takayuki wrote:

* In this incident, autovacuum worker misjudged that
pg_temp_3.fetchchunks can't be deleted, although the creator
(pg_rewind) is no longer active. How can we delete orphan temporary
tables safely?

As long as Postgres sees that its temporary schema is in use, it would think
that the table is not orphaned. Another thing possible would be to have
the session now holding this schema space to reuse fetchchunks so as things
are reset.

I understood you suggested a new session which recycle the temp schema
should erase the zombie metadata of old temp tables or recreate the
temp schema. That sounds easy.

If the new session makes use of the same temporary schema where the
orphan table is, cleanup is possible. Now you have a problem if this is
not available as this depends on the backend ID uniquely assigned. It
would be better to just drop the table manually at the end.
--
Michael

#5Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#4)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Michael Paquier [mailto:michael.paquier@gmail.com]

On Thu, Jan 25, 2018 at 08:10:00AM +0000, Tsunakawa, Takayuki wrote:

I understood you suggested a new session which recycle the temp schema
should erase the zombie metadata of old temp tables or recreate the
temp schema. That sounds easy.

If the new session makes use of the same temporary schema where the orphan
table is, cleanup is possible. Now you have a problem if this is not available
as this depends on the backend ID uniquely assigned.

Ouch, you're right. If the new session "uses the temp schema," it has a chance to clean the old temp table metadata. However, it won't help if the session doesn't try to use the temp schema by creating a temp table...

It would be better
to just drop the table manually at the end.

Just to solve this very incident, it's so. But this is a bug, so we need to fix it somehow.

Regards
Takayuki Tsunakawa

#6Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Tsunakawa, Takayuki (#1)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Thu, Jan 25, 2018 at 3:14 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

FIX
====================================

I have the following questions. Along which line should I proceed to fix the problem?

* Why does autovacuum launcher always choose only one database when that database need vacuuming for XID wraparound? Shouldn't it also choose other databases?

Yeah, I'd also like to fix this issue. This can be problem even in
other case; there are two databases that require anti-wraparound
vacuum, and one of them has a very large table that could take a long
time to vacuum. In this case, if autovacuum chooses the database
having big table first, another database would not be selected until
an autovacuum worker completes vacuum on the large table. To deal with
it, I think we can make autovacuum workers tell that it is no longer
necessary to launch a new autovacuum worker on the database to
autovacuum launcher before exit. For example, autovacuum worker
reports both the total number of relations and the number of relations
that require an anti-wraparound vacuum to the stats collector.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

#7Robert Haas
robertmhaas@gmail.com
In reply to: Tsunakawa, Takayuki (#1)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Thu, Jan 25, 2018 at 1:14 AM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

* I think temporary tables should not require vacuuming for XID wraparound. Furtherover, should updates/deletes to temporary tables be in-place instead of creating garbage, so that any form of vacuum is unnecessary? Other sessions do not need to read temporary tables.

Temporary tables contain XIDs, so they need to be vacuumed for XID
wraparound. Otherwise, queries against those tables by the session
that created them could yield wrong answers. However, autovacuum
can't perform that vacuuming; it would have to be done by the session.
I think we should consider having backends try to remove their
temporary schema on startup; then, if a temp table in a backend is old
enough that it's due for vacuum for wraparound, have autovacuum kill
the connection. The former is necessary to prevent sessions from
being killed on account of temp tables they "inherited" from a backend
that didn't exit cleanly.

The in-place update idea won't work for a couple of reasons. First, a
command shouldn't see the results of modifications made earlier in the
same command. Second, using cursors, it's possible to have more than
one distinct snapshot open against a temporary table at the same time.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#8Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Masahiko Sawada (#6)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

On Thu, Jan 25, 2018 at 3:14 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

* Why does autovacuum launcher always choose only one database when that

database need vacuuming for XID wraparound? Shouldn't it also choose other
databases?

Yeah, I'd also like to fix this issue. This can be problem even in other
case; there are two databases that require anti-wraparound vacuum, and one
of them has a very large table that could take a long time to vacuum. In
this case, if autovacuum chooses the database having big table first,
another database would not be selected until an autovacuum worker completes
vacuum on the large table. To deal with it, I think we can make autovacuum
workers tell that it is no longer necessary to launch a new autovacuum worker
on the database to autovacuum launcher before exit. For example, autovacuum
worker reports both the total number of relations and the number of relations
that require an anti-wraparound vacuum to the stats collector.

Thanks for commenting. I believe you have deep knowledge and experience with vacuum because you did a great work for freeze map in 9.6, so I appreciate your help!

How would you use those two counts?

How about just modifying do_start_worker(), so that the launcher chooses a database in the following order?

1. wraparound-risky database not being vacuumed by any worker
2. non-wraparound-risky database not being vacuumed by any worker
3. wraparound-risky database being vacuumed by any worker
4. non-wraparound-risky database being vacuumed by any worker

Regards
Takayuki Tsunakawa

#9Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Robert Haas (#7)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Robert Haas [mailto:robertmhaas@gmail.com]

I think we should consider having backends try to remove their temporary
schema on startup; then, if a temp table in a backend is old enough that
it's due for vacuum for wraparound, have autovacuum kill the connection.
The former is necessary to prevent sessions from being killed on account
of temp tables they "inherited" from a backend that didn't exit cleanly.

That seems to be the only reasonable solution. One might feel it annoying to emit WAL during connection establishment to delete the temp schema, but even now the client authentication can emit WAL for hot pruning while scanning pg_database, pg_authid, etc. Thanks.

The in-place update idea won't work for a couple of reasons. First, a
command shouldn't see the results of modifications made earlier in the same
command. Second, using cursors, it's possible to have more than one
distinct snapshot open against a temporary table at the same time.

You're right. And if the transaction rolls back, it needs to see the old tuples, which requires undo log.

Regards
Takayuki Tsunakawa

#10Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Tsunakawa, Takayuki (#8)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Fri, Jan 26, 2018 at 2:22 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

On Thu, Jan 25, 2018 at 3:14 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

* Why does autovacuum launcher always choose only one database when that

database need vacuuming for XID wraparound? Shouldn't it also choose other
databases?

Yeah, I'd also like to fix this issue. This can be problem even in other
case; there are two databases that require anti-wraparound vacuum, and one
of them has a very large table that could take a long time to vacuum. In
this case, if autovacuum chooses the database having big table first,
another database would not be selected until an autovacuum worker completes
vacuum on the large table. To deal with it, I think we can make autovacuum
workers tell that it is no longer necessary to launch a new autovacuum worker
on the database to autovacuum launcher before exit. For example, autovacuum
worker reports both the total number of relations and the number of relations
that require an anti-wraparound vacuum to the stats collector.

Thanks for commenting. I believe you have deep knowledge and experience with vacuum because you did a great work for freeze map in 9.6, so I appreciate your help!

Thanks. The most of hackers on the community have deep knowledge, so
I'd like to hear other opinion.

How would you use those two counts?

What I thought is that a worker reports these two values after scanned
pg_class and after freezed a table. The launcher decides to launch a
new worker if the number of tables requiring anti-wraparound vacuum is
greater than the number of workers running on the database. Similarly,
the autovacuum launcher doesn't launch a new worker if two values are
equal, which means all tables requiring an anti-wraparound vacuum is
being vacuumed. There are chances that new relation is added during a
worker is running on the last one table that requires anti-wraparound
vacuum and launcher launches a new worker on the database. I think
it's no problem because the new worker would update that two values
and exits soon.

How about just modifying do_start_worker(), so that the launcher chooses a database in the following order?

1. wraparound-risky database not being vacuumed by any worker
2. non-wraparound-risky database not being vacuumed by any worker
3. wraparound-risky database being vacuumed by any worker
4. non-wraparound-risky database being vacuumed by any worker

IMO the limiting the number of worker on a database to 1 seems risky.
If a database has many tables that require an anti-wraparound vacuum,
it takes a long time to freeze the all of these tables. In current
implementation, as I mentioned as above, launcher can launch multiple
worker on the one database. Since the above idea would be complex a
bit, as an alternated idea it might be better to specify the number of
worker to launch per database by a new GUC parameter or something. If
the number of worker running on the database exceeds that limit, the
launcher doesn't select the database even if the database is about to
wraparound.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

#11Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Masahiko Sawada (#10)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

What I thought is that a worker reports these two values after scanned
pg_class and after freezed a table. The launcher decides to launch a new
worker if the number of tables requiring anti-wraparound vacuum is greater
than the number of workers running on the database. Similarly, the
autovacuum launcher doesn't launch a new worker if two values are equal,
which means all tables requiring an anti-wraparound vacuum is being vacuumed.
There are chances that new relation is added during a worker is running
on the last one table that requires anti-wraparound vacuum and launcher
launches a new worker on the database. I think it's no problem because the
new worker would update that two values and exits soon.

I got it. Currently, the launcher assigns all workers to one database even if that database has only one table in danger of wraparound. With your suggestion, the launcher assigns as many workers as the tables to be frozen, and use remaining workers for the other databases.

How about just modifying do_start_worker(), so that the launcher chooses

a database in the following order?

1. wraparound-risky database not being vacuumed by any worker 2.
non-wraparound-risky database not being vacuumed by any worker 3.
wraparound-risky database being vacuumed by any worker 4.
non-wraparound-risky database being vacuumed by any worker

IMO the limiting the number of worker on a database to 1 seems risky.
If a database has many tables that require an anti-wraparound vacuum, it
takes a long time to freeze the all of these tables. In current
implementation, as I mentioned as above, launcher can launch multiple worker
on the one database.

I can understand your concern. On the other hand, it's unfair that one database could monopolize all workers, because other databases might also be facing wraparound risk.

Since the above idea would be complex a bit, as an
alternated idea it might be better to specify the number of worker to launch
per database by a new GUC parameter or something. If the number of worker
running on the database exceeds that limit, the launcher doesn't select
the database even if the database is about to wraparound.

I'm afraid the GUC would be difficult for the user to understand and tune.

I want to submit the patch for handling the garbage temporary table metadata as Robert suggested in the next CF. That should be enough to prevent this customer's problem. I would appreciate if anyone could address the other improvement that Sawada-san proposed.

Regards
Takayuki Tsunakawa

#12Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Tsunakawa, Takayuki (#11)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Mon, Jan 29, 2018 at 3:33 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

What I thought is that a worker reports these two values after scanned
pg_class and after freezed a table. The launcher decides to launch a new
worker if the number of tables requiring anti-wraparound vacuum is greater
than the number of workers running on the database. Similarly, the
autovacuum launcher doesn't launch a new worker if two values are equal,
which means all tables requiring an anti-wraparound vacuum is being vacuumed.
There are chances that new relation is added during a worker is running
on the last one table that requires anti-wraparound vacuum and launcher
launches a new worker on the database. I think it's no problem because the
new worker would update that two values and exits soon.

I got it. Currently, the launcher assigns all workers to one database even if that database has only one table in danger of wraparound. With your suggestion, the launcher assigns as many workers as the tables to be frozen, and use remaining workers for the other databases.

How about just modifying do_start_worker(), so that the launcher chooses

a database in the following order?

1. wraparound-risky database not being vacuumed by any worker 2.
non-wraparound-risky database not being vacuumed by any worker 3.
wraparound-risky database being vacuumed by any worker 4.
non-wraparound-risky database being vacuumed by any worker

IMO the limiting the number of worker on a database to 1 seems risky.
If a database has many tables that require an anti-wraparound vacuum, it
takes a long time to freeze the all of these tables. In current
implementation, as I mentioned as above, launcher can launch multiple worker
on the one database.

I can understand your concern. On the other hand, it's unfair that one database could monopolize all workers, because other databases might also be facing wraparound risk.

On third thought, we can change the policy of launching workers so
that the launcher dispatches workers evenly to wraparound-risky
databases instead of choosing only one most wraparound-risky database.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

#13Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Masahiko Sawada (#12)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

On Mon, Jan 29, 2018 at 3:33 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

I can understand your concern. On the other hand, it's unfair that one

database could monopolize all workers, because other databases might also
be facing wraparound risk.

On third thought, we can change the policy of launching workers so that
the launcher dispatches workers evenly to wraparound-risky databases
instead of choosing only one most wraparound-risky database.

+1

Regards
Takayuki Tsunakawa

#14Robert Haas
robertmhaas@gmail.com
In reply to: Tsunakawa, Takayuki (#11)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Mon, Jan 29, 2018 at 1:33 AM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

Since the above idea would be complex a bit, as an
alternated idea it might be better to specify the number of worker to launch
per database by a new GUC parameter or something. If the number of worker
running on the database exceeds that limit, the launcher doesn't select
the database even if the database is about to wraparound.

I'm afraid the GUC would be difficult for the user to understand and tune.

I agree. It's autovacuum's job to do the correct thing. If it's not,
then we need figure out how to make it do the right thing. Adding a
GUC seems like saying we don't know what the right thing to do is but
we hope the user does know. That's not a good answer.

Unfortunately, I think a full solution to the problem of allocating AV
workers to avoid wraparound is quite complex. Suppose that database A
will force a shutdown due to impending wraparound in 4 hours and
database B will force a shutdown in 12 hours. On first blush, it
seems that we should favor adding workers to A. But it might be that
database A needs only 2 hours of vacuuming to avoid a shutdown whereas
B needs 12 hours. In that case, it seems that we ought to instead
favor adding workers to B. However, it might be the case that A has
more table coming do for wraparound 6 hours from now, and we need
another 15 hours of vacuuming to avoid that shutdown. That would
favor adding workers to A. Then again, it might be that A and B
already both have workers, and that adding yet another worker to A
won't speed anything up (because only large tables remain to be
processed and each has a worker already), whereas adding a worker to B
would speed things up (because it still has a big table that we could
immediately start to vacuum for wraparound). In that case, perhaps we
ought to instead add a worker to B. But, thinking some more, it seems
like that should cause autovacuum_vacuum_cost_limit to be reallocated
among the workers, making the existing vacuum processes take longer,
which might actually make a bad situation worse. It seems possible
that the right answer could be to start no new autovacuum worker at
all.

Given all of the foregoing this seems like a very hard problem. I
can't even articulate a clear set of rules for what our priorities
should be, and it seems that such rules would depend on the rate at
which we're consuming XIDs, how close we are in each database to a
wraparound shutdown, what tables exist in each database, how big the
not-all-frozen part of each one is, how big their indexes are, how
much they're holding back relfrozenxid, and which ones already have
workers, among other things. I think it's quite possible that we can
come up with something that's better than what we have now without
embarking on a huge project, but it's not going to be anywhere near
perfect because this is really complicated, and there's a real risk
that we'll just making some cases better and others worse rather than
actually coming out ahead overall.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#15Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Robert Haas (#14)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Robert Haas [mailto:robertmhaas@gmail.com]

Unfortunately, I think a full solution to the problem of allocating AV
workers to avoid wraparound is quite complex.

Yes, that easily puts my small brain into an infinite loop...

Given all of the foregoing this seems like a very hard problem. I can't
even articulate a clear set of rules for what our priorities should be,
and it seems that such rules would depend on the rate at which we're consuming
XIDs, how close we are in each database to a wraparound shutdown, what tables
exist in each database, how big the not-all-frozen part of each one is,
how big their indexes are, how much they're holding back relfrozenxid, and
which ones already have workers, among other things. I think it's quite
possible that we can come up with something that's better than what we have
now without embarking on a huge project, but it's not going to be anywhere
near perfect because this is really complicated, and there's a real risk
that we'll just making some cases better and others worse rather than
actually coming out ahead overall.

So a simple improvement would be to assign workers fairly to databases facing a wraparound risk, as Sawada-san suggested.

One ultimate solution should be the undo-based MVCC that makes vacuuming unnecessary, which you proposed about a year ago...

Regards
Takayuki Tsunakawa

#16Robert Haas
robertmhaas@gmail.com
In reply to: Tsunakawa, Takayuki (#15)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Tue, Jan 30, 2018 at 7:04 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

So a simple improvement would be to assign workers fairly to databases facing a wraparound risk, as Sawada-san suggested.

Is that always an improvement, or does it make some cases better and
others worse?

One ultimate solution should be the undo-based MVCC that makes vacuuming unnecessary, which you proposed about a year ago...

And blogged about yesterday.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#17Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Robert Haas (#16)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Thu, Feb 1, 2018 at 2:01 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Tue, Jan 30, 2018 at 7:04 PM, Tsunakawa, Takayuki
<tsunakawa.takay@jp.fujitsu.com> wrote:

So a simple improvement would be to assign workers fairly to databases facing a wraparound risk, as Sawada-san suggested.

Is that always an improvement, or does it make some cases better and
others worse?

I think the idea would not be an improvement, but just change the
policy. The current launcher's policy is "let's launch a new worker as
much as possible on the database that is at risk of wraparound most".
The idea I suggested makes the cases mentioned on this thread better
while perhaps making other cases worse.

To improve while keeping the current policy, we might want to use the
first idea I proposed. That is, we don't launch a new worker on a
database impending wraparound if the last table of the database is
being vacuumed. But it needs to share new information such as what
tables exist in each database and which tables already have worker. It
might be overkill in order to deal with only such a corner case
though.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

#18Robert Haas
robertmhaas@gmail.com
In reply to: Masahiko Sawada (#17)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Wed, Jan 31, 2018 at 7:37 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I think the idea would not be an improvement, but just change the
policy. The current launcher's policy is "let's launch a new worker as
much as possible on the database that is at risk of wraparound most".
The idea I suggested makes the cases mentioned on this thread better
while perhaps making other cases worse.

To improve while keeping the current policy, we might want to use the
first idea I proposed. That is, we don't launch a new worker on a
database impending wraparound if the last table of the database is
being vacuumed. But it needs to share new information such as what
tables exist in each database and which tables already have worker. It
might be overkill in order to deal with only such a corner case
though.

Something like that might work, but I think it needs more thought.
Maybe, for each database currently being processed by at least 1
worker, advertise in shared memory the oldest XID that isn't already
being vacuumed by some AV worker; when considering which database is
at greatest risk of wraparound, if that value is available, use it
instead of the database's datfrozenxid. Then when all tables that
make that database riskier than some other database already have
workers, some other database can get a chance.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

#19Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Robert Haas (#18)
Re: Temporary tables prevent autovacuum, leading to XID wraparound

On Fri, Feb 2, 2018 at 12:31 AM, Robert Haas <robertmhaas@gmail.com> wrote:

On Wed, Jan 31, 2018 at 7:37 PM, Masahiko Sawada <sawada.mshk@gmail.com> wrote:

I think the idea would not be an improvement, but just change the
policy. The current launcher's policy is "let's launch a new worker as
much as possible on the database that is at risk of wraparound most".
The idea I suggested makes the cases mentioned on this thread better
while perhaps making other cases worse.

To improve while keeping the current policy, we might want to use the
first idea I proposed. That is, we don't launch a new worker on a
database impending wraparound if the last table of the database is
being vacuumed. But it needs to share new information such as what
tables exist in each database and which tables already have worker. It
might be overkill in order to deal with only such a corner case
though.

Something like that might work, but I think it needs more thought.
Maybe, for each database currently being processed by at least 1
worker, advertise in shared memory the oldest XID that isn't already
being vacuumed by some AV worker; when considering which database is
at greatest risk of wraparound, if that value is available, use it
instead of the database's datfrozenxid. Then when all tables that
make that database riskier than some other database already have
workers, some other database can get a chance.

Thank you for suggestion. It sounds more smarter. So it would be more
better if we vacuums database for anti-wraparound in ascending order
of relfrozenxid?

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

#20Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Masahiko Sawada (#19)
RE: Temporary tables prevent autovacuum, leading to XID wraparound

From: Masahiko Sawada [mailto:sawada.mshk@gmail.com]

Thank you for suggestion. It sounds more smarter. So it would be more better
if we vacuums database for anti-wraparound in ascending order of
relfrozenxid?

I thought so, too. The current behavior is inconsistent: the launcher tries to assign all workers to one database with the biggest wraparound risk in order to eliminate the risk as fast as possible, but the workers don't get hurry to reduce the risk.

Regards
Takayuki Tsunakawa

#21Robert Haas
robertmhaas@gmail.com
In reply to: Masahiko Sawada (#19)
#22Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Robert Haas (#7)
#23Robert Haas
robertmhaas@gmail.com
In reply to: Tsunakawa, Takayuki (#22)
#24Michael Paquier
michael@paquier.xyz
In reply to: Robert Haas (#23)
#25Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#24)
#26Michael Paquier
michael@paquier.xyz
In reply to: Tsunakawa, Takayuki (#25)
#27Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#26)
#28Robert Haas
robertmhaas@gmail.com
In reply to: Michael Paquier (#24)
#29Masahiko Sawada
sawada.mshk@gmail.com
In reply to: Robert Haas (#21)
#30Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#23)
#31Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Tom Lane (#30)
#32Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#30)
#33Tom Lane
tgl@sss.pgh.pa.us
In reply to: Robert Haas (#32)
#34Robert Haas
robertmhaas@gmail.com
In reply to: Tom Lane (#33)
#35Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Tom Lane (#30)
#36Michael Paquier
michael@paquier.xyz
In reply to: Tsunakawa, Takayuki (#35)
#37Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#36)
#38Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Tsunakawa, Takayuki (#37)
#39Michael Paquier
michael@paquier.xyz
In reply to: Kyotaro Horiguchi (#38)
#40Tsunakawa, Takayuki
tsunakawa.takay@jp.fujitsu.com
In reply to: Michael Paquier (#39)
#41Michael Paquier
michael@paquier.xyz
In reply to: Tsunakawa, Takayuki (#40)
#42Kyotaro Horiguchi
horikyota.ntt@gmail.com
In reply to: Michael Paquier (#41)
#43Michael Paquier
michael@paquier.xyz
In reply to: Kyotaro Horiguchi (#42)
#44Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#41)
#45Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#43)
#46Tom Lane
tgl@sss.pgh.pa.us
In reply to: Alvaro Herrera (#45)
#47Michael Paquier
michael@paquier.xyz
In reply to: Tom Lane (#46)
#48Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#44)
#49Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#48)
#50Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#49)
#51Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#50)
#52Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#51)
#53Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#52)
#54Andres Freund
andres@anarazel.de
In reply to: Michael Paquier (#52)
#55Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#50)
#56Michael Paquier
michael@paquier.xyz
In reply to: Andres Freund (#54)
#57Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#56)
#58Michael Paquier
michael@paquier.xyz
In reply to: Alvaro Herrera (#57)
#59Michael Paquier
michael@paquier.xyz
In reply to: Alvaro Herrera (#55)
#60Michael Paquier
michael@paquier.xyz
In reply to: Michael Paquier (#59)
#61Alvaro Herrera
alvherre@2ndquadrant.com
In reply to: Michael Paquier (#60)
#62Michael Paquier
michael@paquier.xyz
In reply to: Alvaro Herrera (#61)